syzbot

map[Benign:true Confident:true Explanation:The data race reported by KCSAN occurs on the `parent` field of a `struct maple_node` in `lib/maple_tree.c`. This field is used as a synchronization flag to indicate whether a node is "dead" (i.e., has been removed from the tree and is awaiting RCU-based freeing).

### Analysis

1.  **Mechanism of the Race**:
    *   **Writer (`mas_wmb_replace`)**: When the maple tree is restructured, a writer replaces an old node with one or more new nodes. After the new nodes are linked into the tree, the writer marks the old node as dead by setting its `parent` pointer to point to the node itself (`mte_set_node_dead`). This is followed by an `smp_wmb()` to ensure visibility before the node is scheduled for RCU freeing.
    *   **Reader (`mas_state_walk`)**: Readers walking the tree check if a node is dead using `ma_dead_node`. This function performs an `smp_rmb()` followed by a read of `node->parent`. If the node is dead, the reader restarts the walk from the root.

2.  **Atomicity and Tearing**:
    The `parent` field is a pointer. On all supported Linux kernel architectures, naturally aligned 64-bit (or 32-bit on 32-bit systems) pointer accesses are atomic. Since `struct maple_node` is highly aligned (256 bytes), the `parent` pointer is guaranteed to be aligned. Thus, a reader will always see either the old parent pointer or the new "dead" pointer value, never a torn or garbage value.

3.  **Safety under RCU**:
    The maple tree uses RCU to protect readers. Even if a reader misses the "dead" flag due to the data race and continues to use a stale pointer, the node itself remains valid and its contents (pivots and slots) are consistent until the RCU grace period expires. The reader will eventually reach a leaf or catch the "dead" flag at a subsequent level of the tree. This is standard RCU behavior where readers may see slightly stale but consistent data.

4.  **Robustness of the Logic**:
    The implementation includes explicit checks and memory barriers (`smp_rmb`/`smp_wmb`) to ensure that data read from a node is consistent with its "dead" status. For example, in `mtree_range_walk`, the reader reads a slot and then checks the dead flag. The `smp_rmb` ensures the slot is read before the flag. If the flag indicates the node is not dead, the slot value must have been valid at the time of the read.

### Conclusion

The data race is a consequence of using a plain pointer access for a synchronization flag in a high-performance, lockless data structure. While it violates the formal memory model by not using `READ_ONCE`/`WRITE_ONCE`, it is functionally safe because of the atomicity of pointer accesses and the robust retry logic built into the maple tree's RCU implementation. It does not lead to memory corruption or crashes with a conservative compiler.

Verdict: **Benign**]

map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/83e3e4073a4cd51a6ced0f93cc08b3f6ffb54f19]

map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/b54bee6631687294095c3f2df1d84662f988232e]

map[Index:codesearch-index]

gemini-3-flash-preview

map[Benign:true Confident:true]

You are an experienced Linux kernel developer tasked with determining if the given kernel
data race is benign or not. The data race report is from KCSAN tool.
It contains 2 stack traces of the memory accesses that constitute a data race.

A "benign" data races are on a simple int/bool variable or similar field,
and the accesses are not supposed to be protected by any mutual exclusion primitives.
Common examples of such "benign" data races are accesses to various flags fields,
statistics counters, and similar. A "benign" data race does not lead to memory corruption/crash
with a conservative compiler that compiles memory accesses to primitive types
effectively as atomic.

A non-benign (or "harmful" data race) can lead to corruption/crash even with
a conservative compiler that compiles memory accesses to primitive types
effectively as atomic. A common example of a "harmful" data races is race on
a complex container (list/hashmap/etc), where accesses are supposed to be protected
by a mutual exclusion primitive.

Also consider races that happen at the same time with the given one.
If there is no synchronization in between, other memory accesses in the involved threads
race with each other if they access the same memory. For example, if both threads execute:

	some_struct->have_elements = true;
	list_add(new_node, &some_struct->list_head);

the race on some_struct->have_elements may appear benign, however it also implies there
is a race on some_struct->list_head which is not benign, since the list is not thread-safe.

Take into account that on 32-bit systems 64-bit memory accesses may be split into two accesses,
and thus even with a conservative compiler may not be fully atomic. However, such races may
still be benign depending on how writes are done, and how read data is used.

In the final reply explain why you think the given data race is benign or is harmful.

Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The data race report is:

==================================================================
BUG: KCSAN: data-race in mas_state_walk / mas_wmb_replace

write to 0xffff88819e14cb00 of 8 bytes by task 30712 on cpu 0:
 mte_set_node_dead lib/maple_tree.c:334 [inline]
 mas_put_in_tree lib/maple_tree.c:1565 [inline]
 mas_topiary_replace lib/maple_tree.c:2341 [inline]
 mas_wmb_replace+0x20d/0x1610 lib/maple_tree.c:2433
 mas_split lib/maple_tree.c:3052 [inline]
 mas_commit_b_node lib/maple_tree.c:3072 [inline]
 mas_wr_bnode lib/maple_tree.c:3739 [inline]
 mas_wr_store_entry+0x16b6/0x2ed0 lib/maple_tree.c:3771
 mas_store_prealloc+0x72e/0xa70 lib/maple_tree.c:5169
 vma_iter_store_overwrite mm/vma.h:607 [inline]
 vma_iter_store_new mm/vma.h:614 [inline]
 vma_complete+0x3c7/0x850 mm/vma.c:356
 __split_vma+0x5db/0x640 mm/vma.c:566
 split_vma mm/vma.c:596 [inline]
 vma_modify+0xbc9/0xd00 mm/vma.c:1680
 vma_modify_flags+0x182/0x210 mm/vma.c:1700
 mprotect_fixup+0x312/0x5d0 mm/mprotect.c:756
 do_mprotect_pkey+0x67b/0x920 mm/mprotect.c:930
 __do_sys_mprotect mm/mprotect.c:951 [inline]
 __se_sys_mprotect mm/mprotect.c:948 [inline]
 __x64_sys_mprotect+0x48/0x60 mm/mprotect.c:948
 x64_sys_call+0x2768/0x3020 arch/x86/include/generated/asm/syscalls_64.h:11
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x370 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

read to 0xffff88819e14cb00 of 8 bytes by task 30715 on cpu 1:
 ma_dead_node lib/maple_tree.c:572 [inline]
 mte_dead_node lib/maple_tree.c:587 [inline]
 mas_start lib/maple_tree.c:1207 [inline]
 mas_state_walk+0x119/0x650 lib/maple_tree.c:3291
 mas_walk+0x60/0x150 lib/maple_tree.c:4599
 lock_vma_under_rcu+0xc9/0x210 mm/mmap_lock.c:304
 do_user_addr_fault+0x232/0x1050 arch/x86/mm/fault.c:1325
 handle_page_fault arch/x86/mm/fault.c:1474 [inline]
 exc_page_fault+0x62/0xa0 arch/x86/mm/fault.c:1527
 asm_exc_page_fault+0x26/0x30 arch/x86/include/asm/idtentry.h:618

value changed: 0xffff888104da2f81 -> 0xffff88819e14cb00

Reported by Kernel Concurrency Sanitizer on:
CPU: 1 UID: 0 PID: 30715 Comm: syz.6.15537 Tainted: G        W           syzkaller #0 PREEMPT(full) 
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2026
==================================================================

The data race reported by KCSAN occurs on the `parent` field of a `struct maple_node` in `lib/maple_tree.c`. This field is used as a synchronization flag to indicate whether a node is "dead" (i.e., has been removed from the tree and is awaiting RCU-based freeing).

### Analysis

1.  **Mechanism of the Race**:
    *   **Writer (`mas_wmb_replace`)**: When the maple tree is restructured, a writer replaces an old node with one or more new nodes. After the new nodes are linked into the tree, the writer marks the old node as dead by setting its `parent` pointer to point to the node itself (`mte_set_node_dead`). This is followed by an `smp_wmb()` to ensure visibility before the node is scheduled for RCU freeing.
    *   **Reader (`mas_state_walk`)**: Readers walking the tree check if a node is dead using `ma_dead_node`. This function performs an `smp_rmb()` followed by a read of `node->parent`. If the node is dead, the reader restarts the walk from the root.

2.  **Atomicity and Tearing**:
    The `parent` field is a pointer. On all supported Linux kernel architectures, naturally aligned 64-bit (or 32-bit on 32-bit systems) pointer accesses are atomic. Since `struct maple_node` is highly aligned (256 bytes), the `parent` pointer is guaranteed to be aligned. Thus, a reader will always see either the old parent pointer or the new "dead" pointer value, never a torn or garbage value.

3.  **Safety under RCU**:
    The maple tree uses RCU to protect readers. Even if a reader misses the "dead" flag due to the data race and continues to use a stale pointer, the node itself remains valid and its contents (pivots and slots) are consistent until the RCU grace period expires. The reader will eventually reach a leaf or catch the "dead" flag at a subsequent level of the tree. This is standard RCU behavior where readers may see slightly stale but consistent data.

4.  **Robustness of the Logic**:
    The implementation includes explicit checks and memory barriers (`smp_rmb`/`smp_wmb`) to ensure that data read from a node is consistent with its "dead" status. For example, in `mtree_range_walk`, the reader reads a slot and then checks the dead flag. The `smp_rmb` ensures the slot is read before the flag. If the flag indicates the node is not dead, the slot value must have been valid at the time of the read.

### Conclusion

The data race is a consequence of using a plain pointer access for a synchronization flag in a high-performance, lockless data structure. While it violates the formal memory model by not using `READ_ONCE`/`WRITE_ONCE`, it is functionally safe because of the atomicity of pointer accesses and the robust retry logic built into the maple tree's RCU implementation. It does not lead to memory corruption or crashes with a conservative compiler.

Verdict: **Benign**

gemini-3-flash-preview

							input: 4244
							output: 87
							thoughts: 438
						

**Data Race Investigation: maple_tree.c**

Okay, so the report flags a data race, a classic read-write conflict, specifically between `mas_wmb_replace` (the write) and `mas_state_walk` (the read) within `lib/maple_tree.c`. Let's unpack this...

The write path seems to originate from `mte_set_node_dead`, which is inlined into `mas_put_in_tree`, then `mas_topiary_replace`, finally culminating in the problematic `mas_wmb_replace` function. The read, conversely, comes from `ma_dead_node`, inlined in `mte_dead_node`, inlined in `mas_start` and finally calls `mas_state_walk`. It appears to be a direct data race between those two functions.

The address in question is `0xffff88819e14cb00`. Now, the initial value was `0xffff888104da2f81`, and it's being *overwritten* with the very address itself! That's... unusual. `mte_set_node_dead` is expected to manipulate a bit, but now it's overwriting the whole address. This is a bit fishy. I will need to understand why this would happen.

Time to dive into the source code for `mte_set_node_dead` and `ma_dead_node`.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mte_set_node_dead]

map[SourceCode: 328:	/*
 329:	 * mte_set_node_dead() - Set a maple encoded node as dead.
 330:	 * @mn: The maple encoded node.
 331:	 */
 332:	static inline void mte_set_node_dead(struct maple_enode *mn)
 333:	{
 334:		mte_to_node(mn)->parent = ma_parent_ptr(mte_to_node(mn));
 335:		smp_wmb(); /* Needed for RCU */
 336:	}
 SourceFile:lib/maple_tree.c]

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:ma_dead_node]

map[SourceCode: 560:	/*
 561:	 * ma_dead_node() - check if the @enode is dead.
 562:	 * @enode: The encoded maple node
 563:	 *
 564:	 * Return: true if dead, false otherwise.
 565:	 */
 566:	static __always_inline bool ma_dead_node(const struct maple_node *node)
 567:	{
 568:		struct maple_node *parent;
 569:	
 570:		/* Do not reorder reads from the node prior to the parent check */
 571:		smp_rmb();
 572:		parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK);
 573:		return (parent == node);
 574:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 5523
							output: 42
							thoughts: 51
						

Here's my summary, as the thought process unfolds:

**Digging into `mte_set_node_dead`**

Okay, hold on a second. I'm looking at this `mte_set_node_dead` function, and it's assigning `node->parent` to `ma_parent_ptr(mte_to_node(mn))`. Interesting. I need to understand what's actually happening here. It's setting the parent pointer, but *how*? And, more importantly, *why*? I know this code, so I can see the structure of the objects being manipulated, but I need to ensure this is the correct sequence. 

Time to investigate `ma_parent_ptr`. Let's see what that function is doing; that's the key to understanding how the parent is being set in this specific dead-node scenario.

requested entity does not exist

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:ma_parent_ptr]

map[SourceCode: SourceFile:]

gemini-3-flash-preview

							input: 5826
							output: 21
							thoughts: 22
						

Okay, here's my attempt at summarizing that thought process, tailored for an expert audience:

**Debugging a Potentially Obscure Definition**

Right, so I've hit a snag. The problem I'm looking at isn't immediately obvious from the current context. My gut feeling is that the definition I'm after, the one causing the issue, is hiding away somewhere in a header file, probably as a macro or perhaps even a static inline function. These things can be absolute devils to track down, especially when the codebase is large or has layers of abstraction.

The next logical step is to dive in and perform a focused search for the name of the problematic symbol. Time to fire up the grep, ripgrep or my IDE's search function and start hunting for that sneaky definition. Hopefully, it's not buried too deep in some deeply nested include structure. Let's get searching!

map[Expression:#define ma_parent_ptr]

map[Output:lib/maple_tree.c-91-
lib/maple_tree.c:92:#define ma_parent_ptr(x) ((struct maple_pnode *)(x))
lib/maple_tree.c-93-#define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT)
]

gemini-3-flash-preview

							input: 6151
							output: 40
							thoughts: 185
						

**Analyzing Maple Tree Dead Node Logic**

Okay, so looking at this code, it's pretty straightforward at first glance. `ma_parent_ptr(node)` is essentially just a type cast, right? A simple pointer dereference. And `mte_set_node_dead` is setting the `node->parent` pointer to point back to the node itself, again with a cast. `ma_dead_node` then checks for this self-referential `parent` pointer (after applying a mask). Makes sense so far. It's a classic dead-node marker.

But hold on... I see `ma_dead_node` uses `smp_rmb()` *before* reading `node->parent` and `mte_set_node_dead` uses `smp_wmb()` *after* writing `node->parent`. Aha! The potential data race is clearly on that `node->parent` field. That's the critical shared variable being protected by the memory barriers. It's the standard producer-consumer pattern, the writer being the one marking the node as dead and the reader checking. We've got the classic memory barrier pairing for ordering: `wmb` on the writer side, `rmb` on the reader side.

Now, let's see how this plays out in the `mas_state_walk` context. I see the `unlikely` branch, checking `mas_start(mas)`. Time to dig into that function and understand its implications.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_start]

map[SourceCode:1177:	/*
1178:	 * mas_start() - Sets up maple state for operations.
1179:	 * @mas: The maple state.
1180:	 *
1181:	 * If mas->status == ma_start, then set the min, max and depth to
1182:	 * defaults.
1183:	 *
1184:	 * Return:
1185:	 * - If mas->node is an error or not mas_start, return NULL.
1186:	 * - If it's an empty tree:     NULL & mas->status == ma_none
1187:	 * - If it's a single entry:    The entry & mas->status == ma_root
1188:	 * - If it's a tree:            NULL & mas->status == ma_active
1189:	 */
1190:	static inline struct maple_enode *mas_start(struct ma_state *mas)
1191:	{
1192:		if (likely(mas_is_start(mas))) {
1193:			struct maple_enode *root;
1194:	
1195:			mas->min = 0;
1196:			mas->max = ULONG_MAX;
1197:	
1198:	retry:
1199:			mas->depth = 0;
1200:			root = mas_root(mas);
1201:			/* Tree with nodes */
1202:			if (likely(xa_is_node(root))) {
1203:				mas->depth = 0;
1204:				mas->status = ma_active;
1205:				mas->node = mte_safe_root(root);
1206:				mas->offset = 0;
1207:				if (mte_dead_node(mas->node))
1208:					goto retry;
1209:	
1210:				return NULL;
1211:			}
1212:	
1213:			mas->node = NULL;
1214:			/* empty tree */
1215:			if (unlikely(!root)) {
1216:				mas->status = ma_none;
1217:				mas->offset = MAPLE_NODE_SLOTS;
1218:				return NULL;
1219:			}
1220:	
1221:			/* Single entry tree */
1222:			mas->status = ma_root;
1223:			mas->offset = MAPLE_NODE_SLOTS;
1224:	
1225:			/* Single entry tree. */
1226:			if (mas->index > 0)
1227:				return NULL;
1228:	
1229:			return root;
1230:		}
1231:	
1232:		return NULL;
1233:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 7494
							output: 43
							thoughts: 195
						

**Digging into the Maple Tree Deletion Path**

Okay, so `mas_start` somehow leads to `mte_dead_node` being called, and that in turn calls `ma_dead_node`. This is interesting; I need to understand this call chain. Let's look at the definition of `mte_dead_node`. Ah, here it is:

```c
 582:	static inline bool mte_dead_node(const struct maple_enode *enode)
 583:	{
 584:		struct maple_node *node;
 585:	
 586:		if (mte_is_root(enode))
 587:			return false;
 588:	
 589:		node = mte_to_node(enode);
 590:		return ma_dead_node(node);
 591:	}
```
Okay, so it takes a `maple_enode`, checks if it's the root node using `mte_is_root()`. If it *is* the root, it immediately returns `false`. Otherwise, it converts the `enode` into a regular `maple_node` with `mte_to_node()` and then calls `ma_dead_node` on that `node`. So, the core of whether a node is considered "dead" by `mte_dead_node` is ultimately decided by `ma_dead_node`. This makes sense. I need to understand the meaning of "dead node" in this context. It seems like it's a step in a deletion process. I should probably also check the definition of `mte_is_root()` to be sure, and `mte_to_node`. But for now, this makes the call flow much clearer.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mte_dead_node]

map[SourceCode: 576:	/*
 577:	 * mte_dead_node() - check if the @enode is dead.
 578:	 * @enode: The encoded maple node
 579:	 *
 580:	 * Return: true if dead, false otherwise.
 581:	 */
 582:	static __always_inline bool mte_dead_node(const struct maple_enode *enode)
 583:	{
 584:		struct maple_node *node;
 585:	
 586:		node = mte_to_node(enode);
 587:		return ma_dead_node(node);
 588:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 8362
							output: 43
							thoughts: 1135
						

**Analyzing the Data Race in the Maple Tree Implementation**

Okay, so I'm looking at this potential data race around `node->parent` in the Maple tree code.  `mte_set_node_dead` is writing to it, and `ma_dead_node` is reading from it. KCSAN is flagging this, which makes me think it needs a closer look.

Initially, I'm thinking, this is a flag indicating the death of a node, and the write sets `node->parent` to a specific value, indicating it's dead. The code's using `smp_wmb()` in `mte_set_node_dead`, which is good for RCU, ensuring that the changes before `node->parent` are visible to other CPUs.

Then I see the `ma_dead_node` function.  It has an `smp_rmb()` *before* reading `node->parent`, and that throws me. Usually, I'd expect the `smp_rmb()` *after* reading the flag to make sure I see the updated data.  But here, the `node->parent` *is* the flag.  It's just the *order* of the memory barrier that feels off.  The comment in `ma_dead_node` says "Do not reorder reads from the node prior to the parent check," which made me think the barrier was there to prevent reads from happening *before* the check.

If `ma_dead_node` is there to check if a node is dead before I access the node's contents, that barrier *after* the read of `node->parent` makes much more sense, so I'd make sure subsequent reads from the node are not reordered before the check. But now, it's *before* the read!

Wait, the comment... I re-read the comment in `ma_dead_node`: `/* Do not reorder reads from the node prior to the parent check */`. If it's *before* the read, it's preventing something *before* the call to `ma_dead_node` from being reordered with reading the parent. If `ma_dead_node` is called, a pointer to the node is already available. If I read from the node and *then* check if it's dead, I want to ensure the read didn't happen after the node was marked dead.

Actually, it's the opposite! If the value returned is NOT dead, any *subsequent* reads from the node *should* see the state it was in when it wasn't dead.

Ah, right, RCU. This all clicks now.  A node is being replaced, and it's marked dead. Readers check if a node is dead. `mas_start` finds a node, then checks if it's dead.  If it's dead, it retries. The race is that `node->parent` is being updated without `READ_ONCE` or `WRITE_ONCE`. On 64-bit, pointers are usually atomic, but KCSAN is still complaining.

So, is this benign? If `ma_dead_node` returns a stale value (false when it should be true), `mas_start` might continue with a dead node. That's *bad* because a dead node means it's no longer in the tree, and the contents are invalid. The maple tree handles this with RCU though.

Let's check where `ma_dead_node` is used.  `mas_start`, `mas_descend`, etc. If `ma_dead_node` returns `false` (not dead) but it was just marked dead, the reader *will* continue using the node. Because it's RCU, the node won't be freed until the grace period. The reader will see some state of the node, and it might continue to a child. The child could also be dead! Or the reader could hit a leaf.

Wait, the node is being *removed*. The writer:

1.  Creates new nodes.
2.  Links the new nodes into the tree.
3.  Marks the old node as dead.

Ok, let's trace the write and check the order of operations, especially in `mas_wmb_replace`. I need to figure out *exactly* what's going on here.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_wmb_replace]

map[SourceCode:2421:	/*
2422:	 * mas_wmb_replace() - Write memory barrier and replace
2423:	 * @mas: The maple state
2424:	 * @old_enode: The old maple encoded node that is being replaced.
2425:	 * @new_height: The new height of the tree as a result of the operation
2426:	 *
2427:	 * Updates gap as necessary.
2428:	 */
2429:	static inline void mas_wmb_replace(struct ma_state *mas,
2430:			struct maple_enode *old_enode, unsigned char new_height)
2431:	{
2432:		/* Insert the new data in the tree */
2433:		mas_topiary_replace(mas, old_enode, new_height);
2434:	
2435:		if (mte_is_leaf(mas->node))
2436:			return;
2437:	
2438:		mas_update_gap(mas);
2439:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 10800
							output: 43
							thoughts: 9
						

Okay, here's my interpretation of that thought process, tailored for an expert audience, and written in the first person:

**My Take on `mas_topiary_replace`**

Alright, let's break this down. When I see `mas_topiary_replace`, my immediate thought is a highly specialized application of mass spectrometry, likely in a proteomic context, possibly with a focus on post-translational modifications. The term "topiary" suggests a manipulation or trimming of something, so I'm envisioning scenarios where we're either replacing something on a complex biological structure or perhaps refining a data set, and "mas" suggests there's a reference to a mass spectrometry based method.

My initial hypothesis is this is probably a computational tool or algorithm, designed to handle some specific problem related to mass spec data. I'm guessing it's designed to substitute or correct data based on some underlying criteria. The replacement could relate to:

*   **Isotope patterns:** Perhaps it's dealing with isotopic interference or resolving ambiguous isotope assignments.
*   **PTM identification:** maybe handling ambiguities in the identification of post-translational modifications (PTMs).
*   **Spectral alignment/normalization:** Or, maybe it's doing some work on spectral alignment or peak intensity normalization.
*   **Data filtering:** Perhaps it's a very advanced filter, that corrects the data, not just filters it.

I'm thinking the creators want to correct some underlying error, or enhance data quality somehow. Given the level of specialized jargon, it's likely a relatively niche tool, designed for a specific sub-area within mass spectrometry or proteomics. I need more information to be certain. Is this a custom script, a new module in an existing software suite, or an entirely standalone application? I'd need to see the context or the associated publication to really get the scope of this. I'm guessing it's probably doing some pretty clever things with high-resolution data!

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_topiary_replace]

map[SourceCode:2314:	/*
2315:	 * mas_topiary_replace() - Replace the data with new data, then repair the
2316:	 * parent links within the new tree.  Iterate over the dead sub-tree and collect
2317:	 * the dead subtrees and topiary the nodes that are no longer of use.
2318:	 *
2319:	 * The new tree will have up to three children with the correct parent.  Keep
2320:	 * track of the new entries as they need to be followed to find the next level
2321:	 * of new entries.
2322:	 *
2323:	 * The old tree will have up to three children with the old parent.  Keep track
2324:	 * of the old entries as they may have more nodes below replaced.  Nodes within
2325:	 * [index, last] are dead subtrees, others need to be freed and followed.
2326:	 *
2327:	 * @mas: The maple state pointing at the new data
2328:	 * @old_enode: The maple encoded node being replaced
2329:	 * @new_height: The new height of the tree as a result of the operation
2330:	 *
2331:	 */
2332:	static inline void mas_topiary_replace(struct ma_state *mas,
2333:			struct maple_enode *old_enode, unsigned char new_height)
2334:	{
2335:		struct ma_state tmp[3], tmp_next[3];
2336:		MA_TOPIARY(subtrees, mas->tree);
2337:		bool in_rcu;
2338:		int i, n;
2339:	
2340:		/* Place data in tree & then mark node as old */
2341:		mas_put_in_tree(mas, old_enode, new_height);
2342:	
2343:		/* Update the parent pointers in the tree */
2344:		tmp[0] = *mas;
2345:		tmp[0].offset = 0;
2346:		tmp[1].status = ma_none;
2347:		tmp[2].status = ma_none;
2348:		while (!mte_is_leaf(tmp[0].node)) {
2349:			n = 0;
2350:			for (i = 0; i < 3; i++) {
2351:				if (mas_is_none(&tmp[i]))
2352:					continue;
2353:	
2354:				while (n < 3) {
2355:					if (!mas_find_child(&tmp[i], &tmp_next[n]))
2356:						break;
2357:					n++;
2358:				}
2359:	
2360:				mas_adopt_children(&tmp[i], tmp[i].node);
2361:			}
2362:	
2363:			if (MAS_WARN_ON(mas, n == 0))
2364:				break;
2365:	
2366:			while (n < 3)
2367:				tmp_next[n++].status = ma_none;
2368:	
2369:			for (i = 0; i < 3; i++)
2370:				tmp[i] = tmp_next[i];
2371:		}
2372:	
2373:		/* Collect the old nodes that need to be discarded */
2374:		if (mte_is_leaf(old_enode))
2375:			return mas_free(mas, old_enode);
2376:	
2377:		tmp[0] = *mas;
2378:		tmp[0].offset = 0;
2379:		tmp[0].node = old_enode;
2380:		tmp[1].status = ma_none;
2381:		tmp[2].status = ma_none;
2382:		in_rcu = mt_in_rcu(mas->tree);
2383:		do {
2384:			n = 0;
2385:			for (i = 0; i < 3; i++) {
2386:				if (mas_is_none(&tmp[i]))
2387:					continue;
2388:	
2389:				while (n < 3) {
2390:					if (!mas_find_child(&tmp[i], &tmp_next[n]))
2391:						break;
2392:	
2393:					if ((tmp_next[n].min >= tmp_next->index) &&
2394:					    (tmp_next[n].max <= tmp_next->last)) {
2395:						mat_add(&subtrees, tmp_next[n].node);
2396:						tmp_next[n].status = ma_none;
2397:					} else {
2398:						n++;
2399:					}
2400:				}
2401:			}
2402:	
2403:			if (MAS_WARN_ON(mas, n == 0))
2404:				break;
2405:	
2406:			while (n < 3)
2407:				tmp_next[n++].status = ma_none;
2408:	
2409:			for (i = 0; i < 3; i++) {
2410:				mas_topiary_node(mas, &tmp[i], in_rcu);
2411:				tmp[i] = tmp_next[i];
2412:			}
2413:		} while (!mte_is_leaf(tmp[0].node));
2414:	
2415:		for (i = 0; i < 3; i++)
2416:			mas_topiary_node(mas, &tmp[i], in_rcu);
2417:	
2418:		mas_mat_destroy(mas, &subtrees);
2419:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 12926
							output: 44
							thoughts: 10
						

Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:

**Navigating the `mas_put_in_tree` Implementation**

Alright, let's break down this `mas_put_in_tree` function. I see it's likely responsible for inserting a "message authentication structure" (MAS) into a tree-like data structure, probably a tree designed for storing and retrieving cryptographic keys or authentication data. The naming convention strongly suggests a specialized application, possibly related to message security or key management within a larger system.

The core concern here is the insertion process itself. I need to consider several key aspects:

*   **Tree Structure:** First, I have to understand the specific type of tree being used. Is it a binary tree, a balanced tree like an AVL or red-black tree, or something more specialized? The choice directly impacts insertion performance (logarithmic, linear, etc.) and complexity. The performance requirements of the broader application will be my guide.
*   **MAS Format:** What is the structure of this MAS itself? Does it contain pointers, nested structures, or other data that needs to be handled carefully during insertion? I'll need to know if the MAS has some identifier. Is the insertion based on this identifier or some other key?
*   **Collision Handling:** Depending on the tree and the key used for insertion, collisions (identical keys or identifiers) could be an issue. How is this function designed to handle collisions? Does it overwrite, reject, or use a chaining or probing technique? The collision policy has huge impacts on the integrity of the tree.
*   **Memory Management:** Given that we're inserting a new data structure (the MAS), memory allocation/deallocation are crucial. Is there a pre-allocated memory pool? Are there potential memory leaks? Is the insertion process thread-safe? Concurrent access to such a fundamental function can cause all kinds of errors.
*   **Error Handling:** The code has to have robust error handling. What happens if the memory allocation fails? What if the MAS is invalid, or if the tree is full? The system needs to report failure conditions gracefully.
*   **Concurrency Issues**: The design should handle scenarios where multiple threads or processes might attempt to insert MAS structures concurrently. Without proper synchronization mechanisms (mutexes, semaphores, atomic operations), data corruption is almost certain.
*   **Node Locking:** If it's a concurrent tree, is there a locking mechanism to protect the integrity of individual nodes and the tree structure during insertion? If so, the locking strategy needs to be efficient to minimize contention.

Essentially, I need to thoroughly understand how this function integrates with the overall key management or authentication framework. Is it a low-level primitive, or a higher-level abstraction? My analysis would hinge on the specific requirements, performance goals, and security implications of the system. I would have to dig into the surrounding code and documentation for a full picture.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_put_in_tree]

map[SourceCode:1539:	/*
1540:	 * mas_put_in_tree() - Put a new node in the tree, smp_wmb(), and mark the old
1541:	 * node as dead.
1542:	 * @mas: the maple state with the new node
1543:	 * @old_enode: The old maple encoded node to replace.
1544:	 * @new_height: if we are inserting a root node, update the height of the tree
1545:	 */
1546:	static inline void mas_put_in_tree(struct ma_state *mas,
1547:			struct maple_enode *old_enode, char new_height)
1548:		__must_hold(mas->tree->ma_lock)
1549:	{
1550:		unsigned char offset;
1551:		void __rcu **slots;
1552:	
1553:		if (mte_is_root(mas->node)) {
1554:			mas_mn(mas)->parent = ma_parent_ptr(mas_tree_parent(mas));
1555:			rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
1556:			mt_set_height(mas->tree, new_height);
1557:		} else {
1558:	
1559:			offset = mte_parent_slot(mas->node);
1560:			slots = ma_slots(mte_parent(mas->node),
1561:					 mas_parent_type(mas, mas->node));
1562:			rcu_assign_pointer(slots[offset], mas->node);
1563:		}
1564:	
1565:		mte_set_node_dead(old_enode);
1566:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 14119
							output: 41
							thoughts: 2437
						

**Analyzing the Maple Tree's RCU-Based Concurrency**

Okay, so I'm looking at how the Maple Tree handles node replacement and concurrent access, specifically around the `mas_put_in_tree` function and related functions like `mte_set_node_dead` and `ma_dead_node`. The basic sequence is: a new node is inserted using `rcu_assign_pointer`, and then the old node is marked as "dead" via `mte_set_node_dead`. The crucial aspect here is the ordering: the new node *replaces* the old one in the tree *before* the old node is flagged as dead.

This setup is inherently RCU-like: readers might still be traversing the "old" node during the replacement phase. The key question is whether this is handled correctly. If a reader is in the middle of a traversal, they might encounter a node that has just been replaced. They need to check if it's dead, and if not, keep going with the (now stale) information. Since the old node contents should be consistent within the grace period, there should not be any problems.

So, `mte_set_node_dead` seems to be doing the following: It modifies `node->parent` in the old node to point to itself, which signifies the node is "dead." The crucial thing is the memory barrier (`smp_wmb()`). Where is that barrier? It happens *after* the `parent` pointer modification. That's an important detail. The accompanying function to check for the node being dead is `ma_dead_node`, it takes in the `node`, does `smp_rmb()`, reads the parent and if the parent is the node itself, then the node is dead.

Now, if a reader's `mas_start` gets the old root, it will check if it's dead, retrying if it's dead. If not, it will continue using the old root. That old root still points to the same data, but it's no longer the actual root of the tree, because we're in the process of replacing the root.

Here's the problem: I'm concerned about potential data races, specifically regarding how `node->parent` is accessed. On a 64-bit architecture, an aligned 64-bit write (and read) is generally atomic. But, is it possible to have a compiler split the operation and read a garbage value? KCSAN points out a technical data race there, and, frankly, the Maple Tree docs should address this directly.

If `ma_dead_node` returns the wrong value (due to this potential torn read), we could end up with `parent == node` being true when it shouldn't be, or vice-versa. If it's *incorrectly* true, `mas_start` retries, but if it is *incorrectly* false, we continue with a dead node. I'm not convinced that's safe. While there are a lot of checks, a dead node implies it's being freed, even though its data *should* be valid for the grace period.

However, after looking at the code for the Maple Tree, I don't see any `READ_ONCE`/`WRITE_ONCE` used on `parent`, and that concerns me.

Now, I think I see the key thing: The combination of `smp_wmb()` in `mte_set_node_dead` and `smp_rmb()` in `ma_dead_node` is essentially a classic "flag" pattern where the "flag" is the `parent` pointer itself. The write-side is: write the new data, issue the memory barrier, then set the `parent` pointer to point to the node itself (marking it as dead). The read-side should: *first* issue the memory barrier, *then* read the `parent` pointer.

But, `ma_dead_node` is doing `smp_rmb()` *before* reading `node->parent`, which is okay. If the read is "torn," the retry mechanism will likely work fine. But I think there should be `READ_ONCE`. And I need to re-evaluate the memory barrier on the read side. If `ma_dead_node` returns true, we retry. If it returns false, we continue. This means we must protect any reads that occur *after* the `ma_dead_node()` from that same data race, and an `smp_rmb()` *before* the `parent` pointer read doesn't do that.

So, in `mas_start` we read the root of the tree, then immediately check `mte_dead_node()`. If it's not

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_descend]

map[SourceCode: 946:	/*
 947:	 * mas_descend() - Descend into the slot stored in the ma_state.
 948:	 * @mas: the maple state.
 949:	 *
 950:	 * Note: Not RCU safe, only use in write side or debug code.
 951:	 */
 952:	static inline void mas_descend(struct ma_state *mas)
 953:	{
 954:		enum maple_type type;
 955:		unsigned long *pivots;
 956:		struct maple_node *node;
 957:		void __rcu **slots;
 958:	
 959:		node = mas_mn(mas);
 960:		type = mte_node_type(mas->node);
 961:		pivots = ma_pivots(node, type);
 962:		slots = ma_slots(node, type);
 963:	
 964:		if (mas->offset)
 965:			mas->min = pivots[mas->offset - 1] + 1;
 966:		mas->max = mas_safe_pivot(mas, pivots, mas->offset, type);
 967:		mas->node = mas_slot(mas, slots, mas->offset);
 968:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 18009
							output: 42
							thoughts: 49
						

Okay, let me think this through.

**Analyzing the `mas_state_walk` Function**

Hmm, so `mas_descend` is explicitly flagged as "Not RCU safe," which immediately raises a red flag. But `mas_state_walk`... ah, that's where things get interesting. It calls `mas_start` first, and then it does some other things. So, I need to understand how `mas_start` and those "other things" interact with the data structures involved, especially in the context of RCU. I'm going to dive deeper into `mas_state_walk` to figure out what's going on here. I need to figure out exactly how it handles concurrent access and potential race conditions. Is the "not RCU safe" limitation propagated through the call chain, or is there some other mechanism being used to guarantee safety? This warrants a closer look. Let's see what's actually going on in that function.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_state_walk]

map[SourceCode:3287:	static inline void *mas_state_walk(struct ma_state *mas)
3288:	{
3289:		void *entry;
3290:	
3291:		entry = mas_start(mas);
3292:		if (mas_is_none(mas))
3293:			return NULL;
3294:	
3295:		if (mas_is_ptr(mas))
3296:			return entry;
3297:	
3298:		return mtree_range_walk(mas);
3299:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 18495
							output: 43
							thoughts: 9
						

Okay, here's my attempt at summarizing the thought process, tailored for an expert audience, and written in the first person:

**Deep Dive into `mtree_range_walk`**

Alright, let's break down this `mtree_range_walk` function. It seems like we're dealing with a specific traversal strategy, likely designed for efficient access within an `mtree` structure. My initial hypothesis is that it's meant to navigate the tree, potentially returning elements within a specified range, or perhaps performing an operation on elements within that range. We're talking performance here, so it's got to be optimized for the common case, presumably.

What's the underlying data structure? Is this using a B-tree variant? Perhaps a more specialized structure tailored for geospatial data or some other metric space? Knowing the underlying structure is *key*. I'm immediately trying to infer the properties of that structure from the function name and purpose. `Range` suggests some form of ordering, which is a common property of most index structures, but the tree-like terminology could mean a hierarchical organization. This could also be a clever optimization over a simple sequential walk.

I'm starting to think about potential edge cases. How does this handle an empty range? What happens if the specified range is outside of the actual data bounds? Are there specific error conditions, or does it return an empty set? The details matter; these are the things that distinguish solid code from code that *looks* right.

The term `walk` is interesting. Is this an iterative or recursive process? Considering the word, I'd wager it's iterative, especially for large datasets. This gives it the flexibility to avoid excessive stack depth or recursion overhead. I'm wondering if there are opportunities for parallel processing or vectorization within the walk. We'll need to look at the actual implementation and usage to see if any such optimizations are in play. A range-specific walk could be an excellent candidate for parallelization, depending on how subtrees are handled.

Finally, what's the purpose of returning elements in the selected range? Is it for performing operations on the data within the range? Is it returning key-value pairs? Perhaps it's a generator? The return value is critical for understanding the intent. I'm focusing my attention on how the retrieved data is consumed downstream.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mtree_range_walk]

map[SourceCode:2538:	static inline void *mtree_range_walk(struct ma_state *mas)
2539:	{
2540:		unsigned long *pivots;
2541:		unsigned char offset;
2542:		struct maple_node *node;
2543:		struct maple_enode *next, *last;
2544:		enum maple_type type;
2545:		void __rcu **slots;
2546:		unsigned char end;
2547:		unsigned long max, min;
2548:		unsigned long prev_max, prev_min;
2549:	
2550:		next = mas->node;
2551:		min = mas->min;
2552:		max = mas->max;
2553:		do {
2554:			last = next;
2555:			node = mte_to_node(next);
2556:			type = mte_node_type(next);
2557:			pivots = ma_pivots(node, type);
2558:			end = ma_data_end(node, type, pivots, max);
2559:			prev_min = min;
2560:			prev_max = max;
2561:			if (pivots[0] >= mas->index) {
2562:				offset = 0;
2563:				max = pivots[0];
2564:				goto next;
2565:			}
2566:	
2567:			offset = 1;
2568:			while (offset < end) {
2569:				if (pivots[offset] >= mas->index) {
2570:					max = pivots[offset];
2571:					break;
2572:				}
2573:				offset++;
2574:			}
2575:	
2576:			min = pivots[offset - 1] + 1;
2577:	next:
2578:			slots = ma_slots(node, type);
2579:			next = mt_slot(mas->tree, slots, offset);
2580:			if (unlikely(ma_dead_node(node)))
2581:				goto dead_node;
2582:		} while (!ma_is_leaf(type));
2583:	
2584:		mas->end = end;
2585:		mas->offset = offset;
2586:		mas->index = min;
2587:		mas->last = max;
2588:		mas->min = prev_min;
2589:		mas->max = prev_max;
2590:		mas->node = last;
2591:		return (void *)next;
2592:	
2593:	dead_node:
2594:		mas_reset(mas);
2595:		return NULL;
2596:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 19853
							output: 36
							thoughts: 1645
						

**Analyzing the Maple Tree's Dead Node Handling**

Okay, so I'm digging into this `mtree_range_walk` function and specifically how it handles potentially dead nodes. The code first reads from the node (pivots, slots) *before* checking the `ma_dead_node` flag. This seems a little odd at first. The purpose, I see, is that the `smp_rmb()` in `ma_dead_node` ensures that reads of the node's data are complete *before* checking the "dead" flag. This design guarantees the consistency of the data read from the node if the node isn't marked as dead.

Let me break down the writer and reader interactions. The writer first replaces the node and then marks the old node as dead using `mte_set_node_dead`. This function sets the old node's `parent` pointer to point to itself and issues a `smp_wmb()`. The reader, on the other hand, reads data from the `old_node` and *then* checks `ma_dead_node`, which includes an `smp_rmb()` and reads `old_node->parent`. This is the key. The barrier ensures the reader observes the `parent` pointer update. If the reader sees a non-dead parent, the earlier data reads were consistent.

But KCSAN flags a data race on `node->parent`. While it *should* be benign on x86 because aligned 64-bit writes are atomic, I'm concerned about compiler reordering. The main point is that KCSAN sees the concurrent access without synchronization, which isn't always accounted for by barriers alone.

The tricky part is that `node->parent` serves two purposes. When a node is alive, it holds the parent pointer and metadata bits. When dead, it's set to point to itself. `ma_dead_node` only compares `parent` with `node`. If `node->parent` is torn, the value of `parent` is anything, but it returns true or false, safely.

The race is between the writer (using a lock, marking the node as dead) and the reader (lockless, checking the dead flag). My thinking is, the maple tree is designed for this type of check on the dead flag. However, I think that using `READ_ONCE`/`WRITE_ONCE` might be the best option here, to avoid compiler weirdness and make KCSAN happy.

So, is this harmful? Most kernel devs consider this type of race on a flag "benign" in that it won't lead to incorrect logic if we assume there is no tearing (and pointers are less prone to tearing). But the maple tree is complex and sensitive to ordering! Let's see if there are other reports or fixes for it in the codebase.

Aha! There's been recent discussion of `mte_set_node_dead` on the mailing lists. Let me check the value change in the KCSAN report. The address being written to is `0xffff88819e14cb00`. So, the node itself is `0xffff88819e14cb00`. The old value of the parent was `0xffff888104da2f81`, with low bits set. So, by masking the low bits, we get the real parent pointer. The new value is the node's address. So, now the comparison returns true.

So, the race *is* definitely there. If the reader sees the old value, it proceeds as normal. If it sees the new value, it retries. If the reader sees a torn value... Now on x86_64, a 64-bit aligned write is atomic, so we need to confirm that `node->parent` is aligned. Let's check the struct layout.

map[ContextFile:lib/maple_tree.c Name:maple_node]

map[Fields:[map[Name: OffsetBits:0 SizeBits:2048]]]

gemini-3-flash-preview

							input: 22400
							output: 41
							thoughts: 20
						

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Analyzing the Unhelpful Response**

Well, that response was less than illuminating. It didn't provide any concrete information to address the problem. My immediate thought is to bypass the fluff and get straight to the critical details. To truly understand the underlying issue, I need to zero in on the core data structure involved. My focus now is clear: I'm not looking for generalities; I'm after the precise definition of `struct maple_node`. This structure is likely at the heart of the matter, and understanding its members, how it's used, and its relationship to other structures will provide the critical clues I need to proceed.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:maple_node]

map[SourceCode: 285:	struct maple_node {
 286:		union {
 287:			struct {
 288:				struct maple_pnode *parent;
 289:				void __rcu *slot[MAPLE_NODE_SLOTS];
 290:			};
 291:			struct {
 292:				void *pad;
 293:				struct rcu_head rcu;
 294:				struct maple_enode *piv_parent;
 295:				unsigned char parent_slot;
 296:				enum maple_type type;
 297:				unsigned char slot_len;
 298:				unsigned int ma_flags;
 299:			};
 300:			struct maple_range_64 mr64;
 301:			struct maple_arange_64 ma64;
 302:		};
 303:	};
 SourceFile:include/linux/maple_tree.h]

gemini-3-flash-preview

							input: 22885
							output: 43
							thoughts: 761
						

**Analyzing Maple Tree Node Deletion**

Okay, so I'm looking at how the `struct maple_node` is handled. It's allocated from a slab cache, specifically `maple_node_cache`. The cool thing is that the alignment is set to `sizeof(struct maple_node)`, which is 256 bytes. That means it's inherently 8-byte aligned, which is important. On x86_64, reading and writing that `parent` field should be atomic, no problem there. I was a little concerned about compiler reordering or potential issues with KCSAN, but the presence of `smp_wmb()` and `smp_rmb()` makes me feel better about it.

Then I saw something interesting in `ma_dead_node`. The code does an `smp_rmb()` before reading the `parent` field. I noticed similar code in `mtree_range_walk`, where the `ma_dead_node` check is used. If that function is inlined, then it guarantees that the node data is read before the `parent` check.

So if we have:
1.  Read node's data
2.  `smp_rmb()`
3.  Read `parent`

this guarantees that data read is valid at the time parent is read. If the node is not dead, data is valid. The `smp_rmb()` after the `mt_slot` call *before* checking `parent` ensures that the node is *not* marked dead at the time that `mt_slot` reads it.

Let's think about the write side. If the write does:
1.  `slots[offset] = new_node`
2.  `smp_wmb()`
3.  `old_node->parent = old_node`

And the reader does:
1.  `next = slots[offset]`
2.  `smp_rmb()`
3.  `parent = old_node->parent`

If the reader sees that `parent != old_node`, that seems fine because it means that it did *not* see the new slot data.

The only real issue would be if the reader sees the OLD `parent` but the node IS actually dead and its contents are being reused, that would be bad. But wait, RCU comes to the rescue! RCU ensures that the node is not reused until the grace period ends. So the node contents (pivots, slots) are *not* changed after it's marked dead, except for the `parent` pointer itself and maybe some RCU stuff.

Let me just quickly confirm what other changes happen when a node is marked dead. I'm going to check the `mas_topiary_replace` which then calls `mas_topiary_node` to be sure. Alright, looks good. So that's how it's safe.

map[ContextFile:lib/maple_tree.c IncludeLines:true Name:mas_topiary_node]

map[SourceCode:2292:	/*
2293:	 * mas_topiary_node() - Dispose of a single node
2294:	 * @mas: The maple state for pushing nodes
2295:	 * @in_rcu: If the tree is in rcu mode
2296:	 *
2297:	 * The node will either be RCU freed or pushed back on the maple state.
2298:	 */
2299:	static inline void mas_topiary_node(struct ma_state *mas,
2300:			struct ma_state *tmp_mas, bool in_rcu)
2301:	{
2302:		struct maple_node *tmp;
2303:		struct maple_enode *enode;
2304:	
2305:		if (mas_is_none(tmp_mas))
2306:			return;
2307:	
2308:		enode = tmp_mas->node;
2309:		tmp = mte_to_node(enode);
2310:		mte_set_node_dead(enode);
2311:		ma_free_rcu(tmp);
2312:	}
 SourceFile:lib/maple_tree.c]

gemini-3-flash-preview

							input: 24661
							output: 20
							thoughts: 1101
						

**Analyzing the Maple Tree Dead Node Race Condition**

Okay, so I'm looking at how the Maple Tree handles node deletion and concurrent reads. The code first marks a node as dead using `mte_set_node_dead` and then schedules it for freeing via `ma_free_rcu`, which uses `call_rcu`.  This looks like a standard RCU pattern – mark, then wait for grace period.

The crucial part here is how readers deal with a dead node.  If a reader encounters a node marked dead after `mte_set_node_dead` but before it's actually freed, it can still safely read the current state of that node until the grace period ends. The `ma_dead_node` check is there as a signal to readers that this node is no longer part of the actively maintained tree and to start over from the root for the latest consistent data. If a reader misses the "dead" flag due to a race, it continues walking the "old" data that was valid at some point during the walk, which should eventually result in a leaf value that was valid during the walk.

Initially, I was wondering why this check was needed at all, since standard RCU semantics already handle this. But Maple Tree's structure is a bit more complex. Since you can restructure the tree during a walk, you can end up traversing a subtree that's been disconnected. The `ma_dead_node` check serves as an optimization, preventing the reader from traversing a disconnected dead subtree and ensuring consistency within the walk. The optimization here is useful.

The race condition itself seems relatively benign—missing the flag should not have a big impact in most cases, right? No risk of an infinite loop since the tree is finite.

But can it lead to a crash? The only way I could see that happening is if the `ma_dead_node` check returns a value (wrongly) that triggers an invalid memory access. But it just returns a boolean.

Then I recalled a similar issue was addressed in the past, so let's see if there are any recent commits related to Maple Tree dead node races. I'll search for "maple tree dead node race".

Okay, I see that `ma_dead_node` has been touched before. Now let me look at the code:

```c
static __always_inline bool ma_dead_node(const struct maple_node *node)
{
	struct maple_node *parent;

	/* Do not reorder reads from the node prior to the parent check */
	smp_rmb();
	parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK);
	return (parent == node);
}
```

My initial thought was that the compiler could reorder the `node->parent` read.  The `smp_rmb()` should prevent reordering. I wonder if the compiler could read `node->parent` twice, but since `parent` is a local variable, it seemed unlikely. But wait...

```c
parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK);
return (parent == node);
```

The compiler could optimize it to:

```c
if (((unsigned long) node->parent & ~MAPLE_NODE_MASK) == (unsigned long)node)
```

In this case the compiler could read `node->parent` twice.

Let's say it reads it twice and the first time is the old parent and the second time is the new parent. Then `((old_parent & ~MASK) == node)` would be false. This doesn't seem to cause a problem either.

It could also read `node->parent` in the opposite order, but in RCU that isn't possible, because nodes change from old to new, they don't go backwards.

The real danger of a missing `READ_ONCE` is that the compiler assumes the value doesn't change, but it should be fine here because of `smp_rmb()`.

However, the Linux kernel memory model says that concurrent accesses should use `READ_ONCE`/`WRITE_ONCE` to avoid undefined behavior.

Is this particular race condition known? I'll check recent patches for `lib/maple_tree.c` by using `grepper` and searching for `READ_ONCE` around `parent`.

map[Expression:READ_ONCE.*parent]

map[Output:arch/s390/kvm/gaccess.c=1550=static inline int _gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr,
--
arch/s390/kvm/gaccess.c-1564-			return -EAGAIN;
arch/s390/kvm/gaccess.c:1565:		parent = READ_ONCE(sg->parent);
arch/s390/kvm/gaccess.c-1566-		if (!parent)
--
arch/s390/kvm/gaccess.c-1568-		scoped_guard(spinlock, &parent->children_lock) {
arch/s390/kvm/gaccess.c:1569:			if (READ_ONCE(sg->parent) != parent)
arch/s390/kvm/gaccess.c-1570-				return -EAGAIN;
--
block/blk-iocost.c=1173=static void current_hweight(struct ioc_gq *iocg, u32 *hw_activep, u32 *hw_inusep)
--
block/blk-iocost.c-1200-		struct ioc_gq *child = iocg->ancestors[lvl + 1];
block/blk-iocost.c:1201:		u64 active_sum = READ_ONCE(parent->child_active_sum);
block/blk-iocost.c:1202:		u64 inuse_sum = READ_ONCE(parent->child_inuse_sum);
block/blk-iocost.c-1203-		u32 active = READ_ONCE(child->active);
--
drivers/infiniband/ulp/ipoib/ipoib_main.c=1312=static int ipoib_get_iflink(const struct net_device *dev)
--
drivers/infiniband/ulp/ipoib/ipoib_main.c-1320-	/* child/vlan interface */
drivers/infiniband/ulp/ipoib/ipoib_main.c:1321:	return READ_ONCE(priv->parent->ifindex);
drivers/infiniband/ulp/ipoib/ipoib_main.c-1322-}
--
fs/ceph/mds_client.c=1284=static int __choose_mds(struct ceph_mds_client *mdsc,
--
fs/ceph/mds_client.c-1332-		rcu_read_lock();
fs/ceph/mds_client.c:1333:		parent = READ_ONCE(req->r_dentry->d_parent);
fs/ceph/mds_client.c-1334-		dir = req->r_parent ? : d_inode_rcu(parent);
--
fs/crypto/hooks.c=32=int fscrypt_file_open(struct inode *inode, struct file *filp)
--
fs/crypto/hooks.c-53-	rcu_read_lock();
fs/crypto/hooks.c:54:	dentry_parent = READ_ONCE(dentry->d_parent);
fs/crypto/hooks.c-55-	inode_parent = d_inode_rcu(dentry_parent);
--
fs/d_path.c=104=static int __prepend_path(const struct dentry *dentry, const struct mount *mnt,
--
fs/d_path.c-107-	while (dentry != root->dentry || &mnt->mnt != root->mnt) {
fs/d_path.c:108:		const struct dentry *parent = READ_ONCE(dentry->d_parent);
fs/d_path.c-109-
fs/d_path.c-110-		if (dentry == mnt->mnt.mnt_root) {
fs/d_path.c:111:			struct mount *m = READ_ONCE(mnt->mnt_parent);
fs/d_path.c-112-			struct mnt_namespace *mnt_ns;
--
fs/dcache.c=965=struct dentry *dget_parent(struct dentry *dentry)
--
fs/dcache.c-976-	seq = raw_seqcount_begin(&dentry->d_seq);
fs/dcache.c:977:	ret = READ_ONCE(dentry->d_parent);
fs/dcache.c-978-	gotref = lockref_get_not_zero(&ret->d_lockref);
--
fs/dcache.c=2597=struct dentry *d_alloc_parallel(struct dentry *parent,
--
fs/dcache.c-2648-	hlist_bl_lock(b);
fs/dcache.c:2649:	if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {
fs/dcache.c-2650-		hlist_bl_unlock(b);
--
fs/dcache.c=3212=ino_t d_parent_ino(struct dentry *dentry)
--
fs/dcache.c-3220-		seq = raw_seqcount_begin(&dentry->d_seq);
fs/dcache.c:3221:		parent = READ_ONCE(dentry->d_parent);
fs/dcache.c-3222-		iparent = d_inode_rcu(parent);
--
fs/libfs.c=1805=int generic_ci_d_compare(const struct dentry *dentry, unsigned int len,
--
fs/libfs.c-1824-
fs/libfs.c:1825:	parent = READ_ONCE(dentry->d_parent);
fs/libfs.c:1826:	dir = READ_ONCE(parent->d_inode);
fs/libfs.c-1827-	if (!dir || !IS_CASEFOLDED(dir))
--
fs/namei.c=3768=struct dentry *lock_rename_child(struct dentry *c1, struct dentry *p2)
fs/namei.c-3769-{
fs/namei.c:3770:	if (READ_ONCE(c1->d_parent) == p2) {
fs/namei.c-3771-		/*
--
fs/overlayfs/super.c=83=static int ovl_revalidate_real(struct dentry *d, unsigned int flags, bool weak)
--
fs/overlayfs/super.c-98-		if (flags & LOOKUP_RCU) {
fs/overlayfs/super.c:99:			parent = READ_ONCE(d->d_parent);
fs/overlayfs/super.c-100-			dir = d_inode_rcu(parent);
--
fs/proc/base.c=877=static bool proc_mem_foll_force(struct file *file, struct mm_struct *mm)
--
fs/proc/base.c-889-					READ_ONCE(task->mm) == mm &&
fs/proc/base.c:890:					READ_ONCE(task->parent) == current;
fs/proc/base.c-891-			put_task_struct(task);
--
kernel/pid_sysctl.h=8=static int pid_mfd_noexec_dointvec_minmax(const struct ctl_table *table,
--
kernel/pid_sysctl.h-22-	/* Equivalent to pidns_memfd_noexec_scope(ns). */
kernel/pid_sysctl.h:23:	scope = max(READ_ONCE(ns->memfd_noexec_scope), parent_scope);
kernel/pid_sysctl.h-24-
--
kernel/signal.c=2168=bool do_notify_parent(struct task_struct *tsk, int sig)
--
kernel/signal.c-2191-		 */
kernel/signal.c:2192:		if (tsk->parent_exec_id != READ_ONCE(tsk->parent->self_exec_id))
kernel/signal.c-2193-			sig = SIGCHLD;
--
kernel/time/timer_migration.c=559=static void __walk_groups_from(up_f up, struct tmigr_walk *data,
--
kernel/time/timer_migration.c-572-		 */
kernel/time/timer_migration.c:573:		group = READ_ONCE(group->parent);
kernel/time/timer_migration.c-574-		data->childmask = child->groupmask;
--
lib/vsprintf.c=909=char *dentry_name(char *buf, char *end, const struct dentry *d, struct printf_spec spec,
--
lib/vsprintf.c-931-
lib/vsprintf.c:932:		p = READ_ONCE(d->d_parent);
lib/vsprintf.c-933-		array[i] = READ_ONCE(d->d_name.name);
--
mm/memcontrol.c=3823=mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
--
mm/memcontrol.c-3848-		memcg->memory.track_failcnt = !memcg_on_dfl;
mm/memcontrol.c:3849:		WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable));
mm/memcontrol.c-3850-		page_counter_init(&memcg->kmem, &parent->kmem, false);
--
mm/page_counter.c=424=void page_counter_calculate_protection(struct page_counter *root,
--
mm/page_counter.c-454-			READ_ONCE(counter->min),
mm/page_counter.c:455:			READ_ONCE(parent->emin),
mm/page_counter.c-456-			atomic_long_read(&parent->children_min_usage),
--
mm/page_counter.c-460-			READ_ONCE(counter->low),
mm/page_counter.c:461:			READ_ONCE(parent->elow),
mm/page_counter.c-462-			atomic_long_read(&parent->children_low_usage),
]

gemini-3-flash-preview

							input: 29382
							output: 15
							thoughts: 39
						

Okay, here's my interpretation of those thoughts, presented as my own mental process, with the assumption I'm deeply familiar with the system:

**Digging into the Maple Tree Implementation**

Right, the initial `grep` didn't turn up any hits for `lib/maple_tree.c`. That's a bit odd; I was expecting something there. Time to be more targeted. I'll focus directly on `lib/maple_tree.c` and specifically look for instances of accessing the `parent` field within that file. I need to understand how the tree traversal is being handled, and where a `parent` pointer is being dereferenced can often reveal important clues about the data structures and algorithms. I'll need to remember to check for any indirect accesses, too.

map[Expression:parent]

map[Output:
Full output is too long, showing 500 out of 484850 lines.
Use more precise expression if possible.

[.clang-format=71=ForEachMacros:
 --
 .clang-format-477-  - 'fwnode_for_each_child_node'
 .clang-format:478:  - 'fwnode_for_each_parent_node'
 .clang-format-479-  - 'fwnode_graph_for_each_endpoint'
 --
 CREDITS=1409=E: rgooch@atnf.csiro.au
 CREDITS:1410:D: parent process death signal to children
 CREDITS-1411-D: prctl() syscall
 --
 CREDITS=2200=W: http://www.xos.nl/
 CREDITS:2201:D: IP transparent proxy support
 CREDITS-2202-S: X/OS Experts in Open Systems BV
 --
 Documentation/ABI/removed/dv1394=4=Description:
 --
 Documentation/ABI/removed/dv1394-10-	libiec61883 which are functionally equivalent, support HDV, and
 Documentation/ABI/removed/dv1394:11:	transparently work on top of the newer firewire kernel drivers.
 Documentation/ABI/removed/dv1394-12-
 --
 Documentation/ABI/removed/raw1394=14=Users:
 Documentation/ABI/removed/raw1394:15:	libraw1394 (works with firewire-cdev too, transparent to library ABI
 Documentation/ABI/removed/raw1394-16-	users)
 --
 Documentation/ABI/removed/video1394=15=Users:
 Documentation/ABI/removed/video1394:16:	libdc1394 (works with firewire-cdev too, transparent to library ABI
 Documentation/ABI/removed/video1394-17-	users)
 --
 Documentation/ABI/stable/sysfs-bus-firewire=58=Description:
 --
 Documentation/ABI/stable/sysfs-bus-firewire-66-		rom_index
 Documentation/ABI/stable/sysfs-bus-firewire:67:			Offset of the unit directory within the parent device's
 Documentation/ABI/stable/sysfs-bus-firewire-68-			(node device's) Configuration ROM, in quadlets.
 --
 Documentation/ABI/stable/sysfs-class-infiniband=204=Description:
 Documentation/ABI/stable/sysfs-class-infiniband-205-		The optional "hw_counters" subdirectory can be under either the
 Documentation/ABI/stable/sysfs-class-infiniband:206:		parent device or the port subdirectories or both. If present,
 Documentation/ABI/stable/sysfs-class-infiniband-207-		there are a list of counters provided by the hardware. They may
 --
 Documentation/ABI/testing/configfs-stp-policy=22=Description:
 Documentation/ABI/testing/configfs-stp-policy-23-		STM device to which this policy applies, read only. Same as the
 Documentation/ABI/testing/configfs-stp-policy:24:		<device> component of its parent directory.
 Documentation/ABI/testing/configfs-stp-policy-25-
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=235=Description:	Specific MJPEG format descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-241-		bFormatIndex		unique id for this format descriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:242:					only defined after parent header is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-243-					linked into the streaming class;
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=261=Description:	Specific MJPEG frame descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-264-		bFrameIndex		   unique id for this framedescriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:265:					   only defined after parent format is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-266-					   linked into the streaming header;
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=294=Description:	Specific uncompressed format descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-297-		bFormatIndex		unique id for this format descriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:298:					only defined after parent header is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-299-					linked into the streaming class;
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=320=Description:	Specific uncompressed frame descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-323-		bFrameIndex		   unique id for this framedescriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:324:					   only defined after parent format is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-325-					   linked into the streaming header;
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=353=Description:    Specific framebased format descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-356-                bFormatIndex            unique id for this format descriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:357:                                        only defined after parent header is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-358-                                        linked into the streaming class;
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc=379=Description:    Specific framebased frame descriptors
 --
 Documentation/ABI/testing/configfs-usb-gadget-uvc-382-                bFrameIndex                unique id for this framedescriptor;
 Documentation/ABI/testing/configfs-usb-gadget-uvc:383:                                           only defined after parent format is
 Documentation/ABI/testing/configfs-usb-gadget-uvc-384-                                           linked into the streaming header;
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs:1:What:           /sys/kernel/debug/accel/<parent_device>/addr
 Documentation/ABI/testing/debugfs-driver-habanalabs-2-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=5=Description:    Sets the device address to be used for read or write through
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-10-
 Documentation/ABI/testing/debugfs-driver-habanalabs:11:What:           /sys/kernel/debug/accel/<parent_device>/clk_gate
 Documentation/ABI/testing/debugfs-driver-habanalabs-12-Date:           May 2020
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=15=Description:    This setting is now deprecated as clock gating is handled solely by the f/w
 Documentation/ABI/testing/debugfs-driver-habanalabs-16-
 Documentation/ABI/testing/debugfs-driver-habanalabs:17:What:           /sys/kernel/debug/accel/<parent_device>/command_buffers
 Documentation/ABI/testing/debugfs-driver-habanalabs-18-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=21=Description:    Displays a list with information about the currently allocated
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-23-
 Documentation/ABI/testing/debugfs-driver-habanalabs:24:What:           /sys/kernel/debug/accel/<parent_device>/command_submission
 Documentation/ABI/testing/debugfs-driver-habanalabs-25-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=28=Description:    Displays a list with information about the currently active
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-30-
 Documentation/ABI/testing/debugfs-driver-habanalabs:31:What:           /sys/kernel/debug/accel/<parent_device>/command_submission_jobs
 Documentation/ABI/testing/debugfs-driver-habanalabs-32-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=35=Description:    Displays a list with detailed information about each JOB (CB) of
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-37-
 Documentation/ABI/testing/debugfs-driver-habanalabs:38:What:           /sys/kernel/debug/accel/<parent_device>/data32
 Documentation/ABI/testing/debugfs-driver-habanalabs-39-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=42=Description:    Allows the root user to read or write directly through the
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-52-
 Documentation/ABI/testing/debugfs-driver-habanalabs:53:What:           /sys/kernel/debug/accel/<parent_device>/data64
 Documentation/ABI/testing/debugfs-driver-habanalabs-54-Date:           Jan 2020
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=57=Description:    Allows the root user to read or write 64 bit data directly
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-67-
 Documentation/ABI/testing/debugfs-driver-habanalabs:68:What:           /sys/kernel/debug/accel/<parent_device>/data_dma
 Documentation/ABI/testing/debugfs-driver-habanalabs-69-Date:           Apr 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=72=Description:    Allows the root user to read from the device's internal
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-85-
 Documentation/ABI/testing/debugfs-driver-habanalabs:86:What:           /sys/kernel/debug/accel/<parent_device>/device
 Documentation/ABI/testing/debugfs-driver-habanalabs-87-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=90=Description:    Enables the root user to set the device to specific state.
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-93-
 Documentation/ABI/testing/debugfs-driver-habanalabs:94:What:           /sys/kernel/debug/accel/<parent_device>/device_release_watchdog_timeout
 Documentation/ABI/testing/debugfs-driver-habanalabs-95-Date:           Oct 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=98=Description:    The watchdog timeout value in seconds for a device release upon
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-100-
 Documentation/ABI/testing/debugfs-driver-habanalabs:101:What:           /sys/kernel/debug/accel/<parent_device>/dma_size
 Documentation/ABI/testing/debugfs-driver-habanalabs-102-Date:           Apr 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=105=Description:    Specify the size of the DMA transaction when using DMA to read
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-110-
 Documentation/ABI/testing/debugfs-driver-habanalabs:111:What:           /sys/kernel/debug/accel/<parent_device>/dump_razwi_events
 Documentation/ABI/testing/debugfs-driver-habanalabs-112-Date:           Aug 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=115=Description:    Dumps all razwi events to dmesg if exist.
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-119-
 Documentation/ABI/testing/debugfs-driver-habanalabs:120:What:           /sys/kernel/debug/accel/<parent_device>/dump_security_violations
 Documentation/ABI/testing/debugfs-driver-habanalabs-121-Date:           Jan 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=124=Description:    Dumps all security violations to dmesg. This will also ack
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-127-
 Documentation/ABI/testing/debugfs-driver-habanalabs:128:What:           /sys/kernel/debug/accel/<parent_device>/engines
 Documentation/ABI/testing/debugfs-driver-habanalabs-129-Date:           Jul 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=132=Description:    Displays the status registers values of the device engines and
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-134-
 Documentation/ABI/testing/debugfs-driver-habanalabs:135:What:           /sys/kernel/debug/accel/<parent_device>/i2c_addr
 Documentation/ABI/testing/debugfs-driver-habanalabs-136-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=139=Description:    Sets I2C device address for I2C transaction that is generated
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-142-
 Documentation/ABI/testing/debugfs-driver-habanalabs:143:What:           /sys/kernel/debug/accel/<parent_device>/i2c_bus
 Documentation/ABI/testing/debugfs-driver-habanalabs-144-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=147=Description:    Sets I2C bus address for I2C transaction that is generated by
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-150-
 Documentation/ABI/testing/debugfs-driver-habanalabs:151:What:           /sys/kernel/debug/accel/<parent_device>/i2c_data
 Documentation/ABI/testing/debugfs-driver-habanalabs-152-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=155=Description:    Triggers an I2C transaction that is generated by the device's
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-159-
 Documentation/ABI/testing/debugfs-driver-habanalabs:160:What:           /sys/kernel/debug/accel/<parent_device>/i2c_len
 Documentation/ABI/testing/debugfs-driver-habanalabs-161-Date:           Dec 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=164=Description:    Sets I2C length in bytes for I2C transaction that is generated by
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-167-
 Documentation/ABI/testing/debugfs-driver-habanalabs:168:What:           /sys/kernel/debug/accel/<parent_device>/i2c_reg
 Documentation/ABI/testing/debugfs-driver-habanalabs-169-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=172=Description:    Sets I2C register id for I2C transaction that is generated by
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-175-
 Documentation/ABI/testing/debugfs-driver-habanalabs:176:What:           /sys/kernel/debug/accel/<parent_device>/led0
 Documentation/ABI/testing/debugfs-driver-habanalabs-177-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=180=Description:    Sets the state of the first S/W led on the device, Not available
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-182-
 Documentation/ABI/testing/debugfs-driver-habanalabs:183:What:           /sys/kernel/debug/accel/<parent_device>/led1
 Documentation/ABI/testing/debugfs-driver-habanalabs-184-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=187=Description:    Sets the state of the second S/W led on the device, Not available
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-189-
 Documentation/ABI/testing/debugfs-driver-habanalabs:190:What:           /sys/kernel/debug/accel/<parent_device>/led2
 Documentation/ABI/testing/debugfs-driver-habanalabs-191-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=194=Description:    Sets the state of the third S/W led on the device, Not available
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-196-
 Documentation/ABI/testing/debugfs-driver-habanalabs:197:What:           /sys/kernel/debug/accel/<parent_device>/memory_scrub
 Documentation/ABI/testing/debugfs-driver-habanalabs-198-Date:           May 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=201=Description:    Allows the root user to scrub the dram memory. The scrubbing
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-203-
 Documentation/ABI/testing/debugfs-driver-habanalabs:204:What:           /sys/kernel/debug/accel/<parent_device>/memory_scrub_val
 Documentation/ABI/testing/debugfs-driver-habanalabs-205-Date:           May 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=208=Description:    The value to which the dram will be set to when the user
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-211-
 Documentation/ABI/testing/debugfs-driver-habanalabs:212:What:           /sys/kernel/debug/accel/<parent_device>/mmu
 Documentation/ABI/testing/debugfs-driver-habanalabs-213-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=216=Description:    Displays the hop values and physical address for a given ASID
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-219-                e.g. to display info about VA 0x1000 for ASID 1 you need to do:
 Documentation/ABI/testing/debugfs-driver-habanalabs:220:                echo "1 0x1000" > /sys/kernel/debug/accel/<parent_device>/mmu
 Documentation/ABI/testing/debugfs-driver-habanalabs-221-
 Documentation/ABI/testing/debugfs-driver-habanalabs:222:What:           /sys/kernel/debug/accel/<parent_device>/mmu_error
 Documentation/ABI/testing/debugfs-driver-habanalabs-223-Date:           Mar 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=226=Description:    Check and display page fault or access violation mmu errors for
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-228-                e.g. to display error info for MMU hw cap bit 9, you need to do:
 Documentation/ABI/testing/debugfs-driver-habanalabs:229:                echo "0x200" > /sys/kernel/debug/accel/<parent_device>/mmu_error
 Documentation/ABI/testing/debugfs-driver-habanalabs:230:                cat /sys/kernel/debug/accel/<parent_device>/mmu_error
 Documentation/ABI/testing/debugfs-driver-habanalabs-231-
 Documentation/ABI/testing/debugfs-driver-habanalabs:232:What:           /sys/kernel/debug/accel/<parent_device>/monitor_dump
 Documentation/ABI/testing/debugfs-driver-habanalabs-233-Date:           Mar 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=236=Description:    Allows the root user to dump monitors status from the device's
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-245-
 Documentation/ABI/testing/debugfs-driver-habanalabs:246:What:           /sys/kernel/debug/accel/<parent_device>/monitor_dump_trig
 Documentation/ABI/testing/debugfs-driver-habanalabs-247-Date:           Mar 2022
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=250=Description:    Triggers dump of monitor data. The value to trigger the operation
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-255-
 Documentation/ABI/testing/debugfs-driver-habanalabs:256:What:           /sys/kernel/debug/accel/<parent_device>/server_type
 Documentation/ABI/testing/debugfs-driver-habanalabs-257-Date:           Feb 2024
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=260=Description:    Exposes the device's server type, maps to enum hl_server_type.
 Documentation/ABI/testing/debugfs-driver-habanalabs-261-
 Documentation/ABI/testing/debugfs-driver-habanalabs:262:What:           /sys/kernel/debug/accel/<parent_device>/set_power_state
 Documentation/ABI/testing/debugfs-driver-habanalabs-263-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=266=Description:    Sets the PCI power state. Valid values are "1" for D0 and "2"
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-268-
 Documentation/ABI/testing/debugfs-driver-habanalabs:269:What:           /sys/kernel/debug/accel/<parent_device>/skip_reset_on_timeout
 Documentation/ABI/testing/debugfs-driver-habanalabs-270-Date:           Jun 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=273=Description:    Sets the skip reset on timeout option for the device. Value of
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-276-
 Documentation/ABI/testing/debugfs-driver-habanalabs:277:What:           /sys/kernel/debug/accel/<parent_device>/state_dump
 Documentation/ABI/testing/debugfs-driver-habanalabs-278-Date:           Oct 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=281=Description:    Gets the state dump occurring on a CS timeout or failure.
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-287-
 Documentation/ABI/testing/debugfs-driver-habanalabs:288:What:           /sys/kernel/debug/accel/<parent_device>/stop_on_err
 Documentation/ABI/testing/debugfs-driver-habanalabs-289-Date:           Mar 2020
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=292=Description:    Sets the stop-on_error option for the device engines. Value of
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-295-
 Documentation/ABI/testing/debugfs-driver-habanalabs:296:What:           /sys/kernel/debug/accel/<parent_device>/timeout_locked
 Documentation/ABI/testing/debugfs-driver-habanalabs-297-Date:           Sep 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=300=Description:    Sets the command submission timeout value in seconds.
 Documentation/ABI/testing/debugfs-driver-habanalabs-301-
 Documentation/ABI/testing/debugfs-driver-habanalabs:302:What:           /sys/kernel/debug/accel/<parent_device>/userptr
 Documentation/ABI/testing/debugfs-driver-habanalabs-303-Date:           Jan 2019
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=306=Description:    Displays a list with information about the current user
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-309-
 Documentation/ABI/testing/debugfs-driver-habanalabs:310:What:           /sys/kernel/debug/accel/<parent_device>/userptr_lookup
 Documentation/ABI/testing/debugfs-driver-habanalabs-311-Date:           Oct 2021
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs=314=Description:    Allows to search for specific user pointers (user virtual
 --
 Documentation/ABI/testing/debugfs-driver-habanalabs-317-
 Documentation/ABI/testing/debugfs-driver-habanalabs:318:What:           /sys/kernel/debug/accel/<parent_device>/vm
 Documentation/ABI/testing/debugfs-driver-habanalabs-319-Date:           Jan 2019
 --
 Documentation/ABI/testing/sysfs-ata=10=For each port, a directory ataX is created where X is the ata_port_id of the
 Documentation/ABI/testing/sysfs-ata:11:port. The device parent is the ata host device.
 Documentation/ABI/testing/sysfs-ata-12-
 --
 Documentation/ABI/testing/sysfs-ata=44=of the port. If a link is behind a PM, its name is linkX.Y where X is
 Documentation/ABI/testing/sysfs-ata:45:ata_port_id of the parent port and Y the PM port.
 Documentation/ABI/testing/sysfs-ata-46-
 --
 Documentation/ABI/testing/sysfs-block-bcache=71=Description:
 --
 Documentation/ABI/testing/sysfs-block-bcache-74-		used to buffer writes until it is mostly full, at which point
 Documentation/ABI/testing/sysfs-block-bcache:75:		writes transparently revert to writethrough mode. Intended only
 Documentation/ABI/testing/sysfs-block-bcache-76-		for benchmarking/testing.
 --
 Documentation/ABI/testing/sysfs-bus-cxl=196=Description:
 --
 Documentation/ABI/testing/sysfs-bus-cxl-203-
 Documentation/ABI/testing/sysfs-bus-cxl:204:What:		/sys/bus/cxl/devices/{port,endpoint}X/parent_dport
 Documentation/ABI/testing/sysfs-bus-cxl-205-Date:		January, 2023
 --
 Documentation/ABI/testing/sysfs-bus-cxl=208=Description:
 --
 Documentation/ABI/testing/sysfs-bus-cxl-212-		descendant CXL port (switch or endpoint) is enumerated it is
 Documentation/ABI/testing/sysfs-bus-cxl:213:		useful to know which 'dport' object in the parent CXL port
 Documentation/ABI/testing/sysfs-bus-cxl:214:		routes to this descendant. The 'parent_dport' symlink points to
 Documentation/ABI/testing/sysfs-bus-cxl-215-		the device representing the downstream port of a CXL switch that
 --
 Documentation/ABI/testing/sysfs-bus-cxl=489=Description:
 --
 Documentation/ABI/testing/sysfs-bus-cxl-491-		When written trigger the driver to allocate space out of the
 Documentation/ABI/testing/sysfs-bus-cxl:492:		parent root decoder's address space. When read the size of the
 Documentation/ABI/testing/sysfs-bus-cxl-493-		address space is reported and should match the span of the
 --
 Documentation/ABI/testing/sysfs-bus-i2c-devices-pca954x=5=Description:
 --
 Documentation/ABI/testing/sysfs-bus-i2c-devices-pca954x-13-			     with an address that conflicts with another
 Documentation/ABI/testing/sysfs-bus-i2c-devices-pca954x:14:			     device on another mux on the same parent bus.
 Documentation/ABI/testing/sysfs-bus-i2c-devices-pca954x-15-		-1	     leave the mux as-is, which is the most optimal
 --
 Documentation/ABI/testing/sysfs-bus-iio=180=What:		/sys/bus/iio/devices/iio:deviceX/in_powerY_reactive_raw
 Documentation/ABI/testing/sysfs-bus-iio:181:What:		/sys/bus/iio/devices/iio:deviceX/in_powerY_apparent_raw
 Documentation/ABI/testing/sysfs-bus-iio-182-KernelVersion:	4.5
 --
 Documentation/ABI/testing/sysfs-bus-iio=193=Description:
 Documentation/ABI/testing/sysfs-bus-iio-194-		Power factor measurement from channel Y. Power factor is the
 Documentation/ABI/testing/sysfs-bus-iio:195:		ratio of active power to apparent power. The value is unitless.
 Documentation/ABI/testing/sysfs-bus-iio-196-
 --
 Documentation/ABI/testing/sysfs-bus-iio=1607=What:		/sys/.../iio:deviceX/in_energyY_reactive_raw
 Documentation/ABI/testing/sysfs-bus-iio:1608:What:		/sys/.../iio:deviceX/in_energyY_apparent_raw
 Documentation/ABI/testing/sysfs-bus-iio-1609-KernelVersion:	4.0
 --
 Documentation/ABI/testing/sysfs-bus-nfit=18=Description:
 Documentation/ABI/testing/sysfs-bus-nfit-19-		(RO) The address (given by the _ADR object) of the device on its
 Documentation/ABI/testing/sysfs-bus-nfit:20:		parent bus of the NVDIMM device containing the NVDIMM region.
 Documentation/ABI/testing/sysfs-bus-nfit-21-
 --
 Documentation/ABI/testing/sysfs-bus-nvdimm=57=Description:	(RO) Shows the CXL bridge device that ties to a CXL memory device
 Documentation/ABI/testing/sysfs-bus-nvdimm:58:		to this NVDIMM device. I.e. the parent of the device returned is
 Documentation/ABI/testing/sysfs-bus-nvdimm-59-		a /sys/bus/cxl/devices/memX instance.
 --
 Documentation/ABI/testing/sysfs-bus-pci=133=Description:
 Documentation/ABI/testing/sysfs-bus-pci-134-		Writing a non-zero value to this attribute will
 Documentation/ABI/testing/sysfs-bus-pci:135:		force a rescan of the device's parent bus and all
 Documentation/ABI/testing/sysfs-bus-pci-136-		child buses, and re-discover devices removed earlier
 --
 Documentation/ABI/testing/sysfs-bus-pci=485=Description:
 --
 Documentation/ABI/testing/sysfs-bus-pci-508-		be unbound from the device and peer devices under the same
 Documentation/ABI/testing/sysfs-bus-pci:509:		parent bridge may need to be soft removed.  In the case of
 Documentation/ABI/testing/sysfs-bus-pci-510-		VGA devices, writing a resize value will remove low level
 --
 Documentation/ABI/testing/sysfs-bus-rbd=126=Description:
 --
 Documentation/ABI/testing/sysfs-bus-rbd-135-
 Documentation/ABI/testing/sysfs-bus-rbd:136:What:		/sys/bus/rbd/devices/<dev-id>/parent
 Documentation/ABI/testing/sysfs-bus-rbd-137-Date:		Nov, 2012
 --
 Documentation/ABI/testing/sysfs-bus-rbd=140=Description:
 Documentation/ABI/testing/sysfs-bus-rbd:141:		(RO) Information identifying the chain of parent images in a
 Documentation/ABI/testing/sysfs-bus-rbd-142-		layered rbd image. Entries are separated by empty lines.
 --
 Documentation/ABI/testing/sysfs-bus-vfio-mdev=27=Description:
 Documentation/ABI/testing/sysfs-bus-vfio-mdev-28-		Writing UUID to this file will create mediated device of
 Documentation/ABI/testing/sysfs-bus-vfio-mdev:29:		type <type-id> for parent device <device>. This is a
 Documentation/ABI/testing/sysfs-bus-vfio-mdev-30-		write-only file.
 --
 Documentation/ABI/testing/sysfs-class-intel_pmt=35=Description:
 Documentation/ABI/testing/sysfs-class-intel_pmt-36-		(RO) The GUID for this telemetry device. The GUID identifies
 Documentation/ABI/testing/sysfs-class-intel_pmt:37:		the version of the XML file for the parent device that is to
 Documentation/ABI/testing/sysfs-class-intel_pmt-38-		be used to get the register layout.
 --
 Documentation/ABI/testing/sysfs-class-intel_pmt=60=Description:
 --
 Documentation/ABI/testing/sysfs-class-intel_pmt-66-		can be determined from an XML file of specified GUID for the
 Documentation/ABI/testing/sysfs-class-intel_pmt:67:		parent device.
 Documentation/ABI/testing/sysfs-class-intel_pmt-68-
 --
 Documentation/ABI/testing/sysfs-class-intel_pmt=81=Description:
 Documentation/ABI/testing/sysfs-class-intel_pmt-82-		(RO) The GUID for this crashlog device. The GUID identifies the
 Documentation/ABI/testing/sysfs-class-intel_pmt:83:		version of the XML file for the parent device that should be
 Documentation/ABI/testing/sysfs-class-intel_pmt-84-		used to determine the register layout.
 --
 Documentation/ABI/testing/sysfs-class-mtd=230=Description:
 Documentation/ABI/testing/sysfs-class-mtd-231-		For a partition, the offset of that partition from the start
 Documentation/ABI/testing/sysfs-class-mtd:232:		of the parent (another partition or a flash device) in bytes.
 Documentation/ABI/testing/sysfs-class-mtd-233-		This attribute is absent on flash devices, so it can be used
 --
 Documentation/ABI/testing/sysfs-class-net=100=Description:
 --
 Documentation/ABI/testing/sysfs-class-net-103-		stacked (e.g: VLAN interfaces) but still have the same MAC
 Documentation/ABI/testing/sysfs-class-net:104:		address as their parent device.
 Documentation/ABI/testing/sysfs-class-net-105-
 --
 Documentation/ABI/testing/sysfs-class-powercap=41=Description:
 --
 Documentation/ABI/testing/sysfs-class-powercap-43-		power zones provide monitoring and control for a subset of
 Documentation/ABI/testing/sysfs-class-powercap:44:		devices under the parent. For example, if there is a parent
 Documentation/ABI/testing/sysfs-class-powercap-45-		power zone for a whole CPU package, each CPU core in it can
 --
 Documentation/ABI/testing/sysfs-class-regulator=233=Description:
 --
 Documentation/ABI/testing/sysfs-class-regulator-239-
 Documentation/ABI/testing/sysfs-class-regulator:240:What:		/sys/class/regulator/.../parent
 Documentation/ABI/testing/sysfs-class-regulator-241-Date:		April 2008
 --
 Documentation/ABI/testing/sysfs-class-regulator=244=Description:
 Documentation/ABI/testing/sysfs-class-regulator:245:		Some regulator directories will contain a link called parent.
 Documentation/ABI/testing/sysfs-class-regulator:246:		This points to the parent or supply regulator if one exists.
 Documentation/ABI/testing/sysfs-class-regulator-247-
 --
 Documentation/ABI/testing/sysfs-devices-pci-host-bridge=4=Description:
 Documentation/ABI/testing/sysfs-devices-pci-host-bridge:5:		A PCI host bridge device parents a PCI bus device topology. PCI
 Documentation/ABI/testing/sysfs-devices-pci-host-bridge:6:		controllers may also parent host bridges. The DDDD:BB format
 Documentation/ABI/testing/sysfs-devices-pci-host-bridge-7-		conveys the PCI domain (ACPI segment) number and root bus number
 --
 Documentation/ABI/testing/sysfs-devices-platform-sh_mobile_lcdc_fb=4=Description:
 --
 Documentation/ABI/testing/sysfs-devices-platform-sh_mobile_lcdc_fb-8-		Stores the alpha blending value for the overlay. Values range
 Documentation/ABI/testing/sysfs-devices-platform-sh_mobile_lcdc_fb:9:		from 0 (transparent) to 255 (opaque). The value is ignored if
 Documentation/ABI/testing/sysfs-devices-platform-sh_mobile_lcdc_fb-10-		the mode is not set to Alpha Blending.
 --
 Documentation/ABI/testing/sysfs-driver-ccp=41=Description:
 Documentation/ABI/testing/sysfs-driver-ccp-42-		The /sys/bus/pci/devices/<BDF>/tsme_status file reports
 Documentation/ABI/testing/sysfs-driver-ccp:43:		the status of transparent secure memory encryption on AMD systems.
 Documentation/ABI/testing/sysfs-driver-ccp-44-		Possible values:
 --
 Documentation/ABI/testing/sysfs-driver-habanalabs=156=Description:    Displays the device's module id
 Documentation/ABI/testing/sysfs-driver-habanalabs-157-
 Documentation/ABI/testing/sysfs-driver-habanalabs:158:What:           /sys/class/accel/accel<n>/device/parent_device
 Documentation/ABI/testing/sysfs-driver-habanalabs-159-Date:           Nov 2023
 --
 Documentation/ABI/testing/sysfs-driver-habanalabs=161=Contact:        ttayar@habana.ai
 Documentation/ABI/testing/sysfs-driver-habanalabs:162:Description:    Displays the name of the parent device of the accel device
 Documentation/ABI/testing/sysfs-driver-habanalabs-163-
 --
 Documentation/ABI/testing/sysfs-edac-ecs=5=Description:
 --
 Documentation/ABI/testing/sysfs-edac-ecs-12-
 Documentation/ABI/testing/sysfs-edac-ecs:13:		The sysfs ECS attr nodes are only present if the parent
 Documentation/ABI/testing/sysfs-edac-ecs-14-		driver has implemented the corresponding attr callback
 --
 Documentation/ABI/testing/sysfs-edac-memory-repair=5=Description:
 --
 Documentation/ABI/testing/sysfs-edac-memory-repair-22-		The sysfs attributes nodes for a repair feature are only
 Documentation/ABI/testing/sysfs-edac-memory-repair:23:		present if the parent driver has implemented the corresponding
 Documentation/ABI/testing/sysfs-edac-memory-repair-24-		attr callback function and provided the necessary operations
 --
 Documentation/ABI/testing/sysfs-edac-scrub=5=Description:
 --
 Documentation/ABI/testing/sysfs-edac-scrub-11-
 Documentation/ABI/testing/sysfs-edac-scrub:12:		The sysfs scrub attr nodes are only present if the parent
 Documentation/ABI/testing/sysfs-edac-scrub-13-		driver has implemented the corresponding attr callback
 --
 Documentation/ABI/testing/sysfs-firmware-dmi-entries=4=Description:
 --
 Documentation/ABI/testing/sysfs-firmware-dmi-entries-74-			  given type.  This value is the same as found
 Documentation/ABI/testing/sysfs-firmware-dmi-entries:75:			  in the parent directory name.
 Documentation/ABI/testing/sysfs-firmware-dmi-entries-76-		position  The ordinal position (zero-based) of the entry
 --
 Documentation/ABI/testing/sysfs-firmware-qemu_fw_cfg=4=Description:
 --
 Documentation/ABI/testing/sysfs-firmware-qemu_fw_cfg-55-			  fw_cfg directory. This value is the same as used in
 Documentation/ABI/testing/sysfs-firmware-qemu_fw_cfg:56:			  the parent directory name.
 Documentation/ABI/testing/sysfs-firmware-qemu_fw_cfg-57-		raw	  The raw bytes of the blob, obtained by selecting the
 --
 Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage:1:What:		/sys/kernel/mm/transparent_hugepage/
 Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage-2-Date:		April 2024
 --
 Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage=4=Description:
 Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage:5:		/sys/kernel/mm/transparent_hugepage/ contains a number of files and
 Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage-6-		subdirectories,
 --
 Documentation/PCI/endpoint/pci-endpoint-cfs.rst=78=If an EPF device has to be associated with 2 EPCs (like in the case of
 Documentation/PCI/endpoint/pci-endpoint-cfs.rst:79:Non-transparent bridge), symlink of endpoint controller connected to primary
 Documentation/PCI/endpoint/pci-endpoint-cfs.rst-80-interface should be added in 'primary' directory and symlink of endpoint
 --
 Documentation/PCI/endpoint/pci-ntb-function.rst=4=PCI NTB Function
 --
]
]

gemini-3-flash-preview

							input: 40436
							output: 26
							thoughts: 78
						

Okay, here's my thought process on this debugging task:

**Refining the Search Strategy**

Alright, that initial `grep` I ran was just too general. It likely returned a ton of irrelevant results. My gut tells me I need to be more precise. Let's revisit the search for those functions, `mte_set_node_dead` and `ma_dead_node`. This time, I'll be more focused on their specific usage. I'll need to use some more targeted search terms or patterns to narrow down the results, maybe use a more sophisticated `grep` and figure out how they are truly used. I really need to understand the dependencies around these functions.

A quick side thought, before diving back into `grep`: It strikes me that the access patterns for the `parent` pointer in this part of the code could be suspect. Maybe some use of `READ_ONCE` or `WRITE_ONCE` could be relevant here, so let me quickly scan `lib/maple_tree.c`. It's possible memory ordering issues are at play, and that might be a clue that these functions are actually fine.

map[Expression:(READ_ONCE|WRITE_ONCE).*parent]

map[Output:arch/s390/kvm/gaccess.c=1550=static inline int _gaccess_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, gpa_t saddr,
--
arch/s390/kvm/gaccess.c-1564-			return -EAGAIN;
arch/s390/kvm/gaccess.c:1565:		parent = READ_ONCE(sg->parent);
arch/s390/kvm/gaccess.c-1566-		if (!parent)
--
arch/s390/kvm/gaccess.c-1568-		scoped_guard(spinlock, &parent->children_lock) {
arch/s390/kvm/gaccess.c:1569:			if (READ_ONCE(sg->parent) != parent)
arch/s390/kvm/gaccess.c-1570-				return -EAGAIN;
--
block/blk-iocost.c=1173=static void current_hweight(struct ioc_gq *iocg, u32 *hw_activep, u32 *hw_inusep)
--
block/blk-iocost.c-1200-		struct ioc_gq *child = iocg->ancestors[lvl + 1];
block/blk-iocost.c:1201:		u64 active_sum = READ_ONCE(parent->child_active_sum);
block/blk-iocost.c:1202:		u64 inuse_sum = READ_ONCE(parent->child_inuse_sum);
block/blk-iocost.c-1203-		u32 active = READ_ONCE(child->active);
--
drivers/infiniband/ulp/ipoib/ipoib_main.c=1312=static int ipoib_get_iflink(const struct net_device *dev)
--
drivers/infiniband/ulp/ipoib/ipoib_main.c-1320-	/* child/vlan interface */
drivers/infiniband/ulp/ipoib/ipoib_main.c:1321:	return READ_ONCE(priv->parent->ifindex);
drivers/infiniband/ulp/ipoib/ipoib_main.c-1322-}
--
drivers/net/ethernet/marvell/octeontx2/nic/qos.c=1608=static int otx2_qos_leaf_del_last(struct otx2_nic *pfvf, u16 classid, bool force,
--
drivers/net/ethernet/marvell/octeontx2/nic/qos.c-1666-	}
drivers/net/ethernet/marvell/octeontx2/nic/qos.c:1667:	WRITE_ONCE(parent->qid, qid);
drivers/net/ethernet/marvell/octeontx2/nic/qos.c-1668-	__set_bit(qid, pfvf->qos.qos_sq_bmap);
--
fs/ceph/mds_client.c=1284=static int __choose_mds(struct ceph_mds_client *mdsc,
--
fs/ceph/mds_client.c-1332-		rcu_read_lock();
fs/ceph/mds_client.c:1333:		parent = READ_ONCE(req->r_dentry->d_parent);
fs/ceph/mds_client.c-1334-		dir = req->r_parent ? : d_inode_rcu(parent);
--
fs/crypto/hooks.c=32=int fscrypt_file_open(struct inode *inode, struct file *filp)
--
fs/crypto/hooks.c-53-	rcu_read_lock();
fs/crypto/hooks.c:54:	dentry_parent = READ_ONCE(dentry->d_parent);
fs/crypto/hooks.c-55-	inode_parent = d_inode_rcu(dentry_parent);
--
fs/d_path.c=104=static int __prepend_path(const struct dentry *dentry, const struct mount *mnt,
--
fs/d_path.c-107-	while (dentry != root->dentry || &mnt->mnt != root->mnt) {
fs/d_path.c:108:		const struct dentry *parent = READ_ONCE(dentry->d_parent);
fs/d_path.c-109-
fs/d_path.c-110-		if (dentry == mnt->mnt.mnt_root) {
fs/d_path.c:111:			struct mount *m = READ_ONCE(mnt->mnt_parent);
fs/d_path.c-112-			struct mnt_namespace *mnt_ns;
--
fs/dcache.c=965=struct dentry *dget_parent(struct dentry *dentry)
--
fs/dcache.c-976-	seq = raw_seqcount_begin(&dentry->d_seq);
fs/dcache.c:977:	ret = READ_ONCE(dentry->d_parent);
fs/dcache.c-978-	gotref = lockref_get_not_zero(&ret->d_lockref);
--
fs/dcache.c=2597=struct dentry *d_alloc_parallel(struct dentry *parent,
--
fs/dcache.c-2648-	hlist_bl_lock(b);
fs/dcache.c:2649:	if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {
fs/dcache.c-2650-		hlist_bl_unlock(b);
--
fs/dcache.c=3212=ino_t d_parent_ino(struct dentry *dentry)
--
fs/dcache.c-3220-		seq = raw_seqcount_begin(&dentry->d_seq);
fs/dcache.c:3221:		parent = READ_ONCE(dentry->d_parent);
fs/dcache.c-3222-		iparent = d_inode_rcu(parent);
--
fs/libfs.c=1805=int generic_ci_d_compare(const struct dentry *dentry, unsigned int len,
--
fs/libfs.c-1824-
fs/libfs.c:1825:	parent = READ_ONCE(dentry->d_parent);
fs/libfs.c:1826:	dir = READ_ONCE(parent->d_inode);
fs/libfs.c-1827-	if (!dir || !IS_CASEFOLDED(dir))
--
fs/namei.c=3768=struct dentry *lock_rename_child(struct dentry *c1, struct dentry *p2)
fs/namei.c-3769-{
fs/namei.c:3770:	if (READ_ONCE(c1->d_parent) == p2) {
fs/namei.c-3771-		/*
--
fs/overlayfs/super.c=83=static int ovl_revalidate_real(struct dentry *d, unsigned int flags, bool weak)
--
fs/overlayfs/super.c-98-		if (flags & LOOKUP_RCU) {
fs/overlayfs/super.c:99:			parent = READ_ONCE(d->d_parent);
fs/overlayfs/super.c-100-			dir = d_inode_rcu(parent);
--
fs/proc/base.c=877=static bool proc_mem_foll_force(struct file *file, struct mm_struct *mm)
--
fs/proc/base.c-889-					READ_ONCE(task->mm) == mm &&
fs/proc/base.c:890:					READ_ONCE(task->parent) == current;
fs/proc/base.c-891-			put_task_struct(task);
--
include/linux/rbtree_augmented.h=195=__rb_change_child(struct rb_node *old, struct rb_node *new,
--
include/linux/rbtree_augmented.h-199-		if (parent->rb_left == old)
include/linux/rbtree_augmented.h:200:			WRITE_ONCE(parent->rb_left, new);
include/linux/rbtree_augmented.h-201-		else
include/linux/rbtree_augmented.h:202:			WRITE_ONCE(parent->rb_right, new);
include/linux/rbtree_augmented.h-203-	} else
--
include/linux/rbtree_augmented.h=224=__rb_erase_augmented(struct rb_node *node, struct rb_root *root,
--
include/linux/rbtree_augmented.h-294-			child2 = successor->rb_right;
include/linux/rbtree_augmented.h:295:			WRITE_ONCE(parent->rb_left, child2);
include/linux/rbtree_augmented.h-296-			WRITE_ONCE(successor->rb_right, child);
--
kernel/pid_sysctl.h=8=static int pid_mfd_noexec_dointvec_minmax(const struct ctl_table *table,
--
kernel/pid_sysctl.h-22-	/* Equivalent to pidns_memfd_noexec_scope(ns). */
kernel/pid_sysctl.h:23:	scope = max(READ_ONCE(ns->memfd_noexec_scope), parent_scope);
kernel/pid_sysctl.h-24-
--
kernel/signal.c=2168=bool do_notify_parent(struct task_struct *tsk, int sig)
--
kernel/signal.c-2191-		 */
kernel/signal.c:2192:		if (tsk->parent_exec_id != READ_ONCE(tsk->parent->self_exec_id))
kernel/signal.c-2193-			sig = SIGCHLD;
--
kernel/time/timer_migration.c=559=static void __walk_groups_from(up_f up, struct tmigr_walk *data,
--
kernel/time/timer_migration.c-572-		 */
kernel/time/timer_migration.c:573:		group = READ_ONCE(group->parent);
kernel/time/timer_migration.c-574-		data->childmask = child->groupmask;
--
lib/rbtree.c=85=__rb_insert(struct rb_node *node, struct rb_root *root,
--
lib/rbtree.c-154-				tmp = node->rb_left;
lib/rbtree.c:155:				WRITE_ONCE(parent->rb_right, tmp);
lib/rbtree.c:156:				WRITE_ONCE(node->rb_left, parent);
lib/rbtree.c-157-				if (tmp)
--
lib/rbtree.c-175-			 */
lib/rbtree.c:176:			WRITE_ONCE(gparent->rb_left, tmp); /* == parent->rb_right */
lib/rbtree.c:177:			WRITE_ONCE(parent->rb_right, gparent);
lib/rbtree.c-178-			if (tmp)
--
lib/rbtree.c-198-				tmp = node->rb_right;
lib/rbtree.c:199:				WRITE_ONCE(parent->rb_left, tmp);
lib/rbtree.c:200:				WRITE_ONCE(node->rb_right, parent);
lib/rbtree.c-201-				if (tmp)
--
lib/rbtree.c-210-			/* Case 3 - left rotate at gparent */
lib/rbtree.c:211:			WRITE_ONCE(gparent->rb_right, tmp); /* == parent->rb_left */
lib/rbtree.c:212:			WRITE_ONCE(parent->rb_left, gparent);
lib/rbtree.c-213-			if (tmp)
--
lib/rbtree.c=227=____rb_erase_color(struct rb_node *parent, struct rb_root *root,
--
lib/rbtree.c-252-				tmp1 = sibling->rb_left;
lib/rbtree.c:253:				WRITE_ONCE(parent->rb_right, tmp1);
lib/rbtree.c:254:				WRITE_ONCE(sibling->rb_left, parent);
lib/rbtree.c-255-				rb_set_parent_color(tmp1, parent, RB_BLACK);
--
lib/rbtree.c-321-				WRITE_ONCE(tmp2->rb_right, sibling);
lib/rbtree.c:322:				WRITE_ONCE(parent->rb_right, tmp2);
lib/rbtree.c-323-				if (tmp1)
--
lib/rbtree.c-342-			tmp2 = sibling->rb_left;
lib/rbtree.c:343:			WRITE_ONCE(parent->rb_right, tmp2);
lib/rbtree.c:344:			WRITE_ONCE(sibling->rb_left, parent);
lib/rbtree.c-345-			rb_set_parent_color(tmp1, sibling, RB_BLACK);
--
lib/rbtree.c-356-				tmp1 = sibling->rb_right;
lib/rbtree.c:357:				WRITE_ONCE(parent->rb_left, tmp1);
lib/rbtree.c:358:				WRITE_ONCE(sibling->rb_right, parent);
lib/rbtree.c-359-				rb_set_parent_color(tmp1, parent, RB_BLACK);
--
lib/rbtree.c-385-				WRITE_ONCE(tmp2->rb_left, sibling);
lib/rbtree.c:386:				WRITE_ONCE(parent->rb_left, tmp2);
lib/rbtree.c-387-				if (tmp1)
--
lib/rbtree.c-395-			tmp2 = sibling->rb_right;
lib/rbtree.c:396:			WRITE_ONCE(parent->rb_left, tmp2);
lib/rbtree.c:397:			WRITE_ONCE(sibling->rb_right, parent);
lib/rbtree.c-398-			rb_set_parent_color(tmp1, sibling, RB_BLACK);
--
lib/vsprintf.c=909=char *dentry_name(char *buf, char *end, const struct dentry *d, struct printf_spec spec,
--
lib/vsprintf.c-931-
lib/vsprintf.c:932:		p = READ_ONCE(d->d_parent);
lib/vsprintf.c-933-		array[i] = READ_ONCE(d->d_name.name);
--
mm/memcontrol.c=209=static void memcg_reparent_objcgs(struct mem_cgroup *memcg,
--
mm/memcontrol.c-221-	list_for_each_entry(iter, &memcg->objcg_list, list)
mm/memcontrol.c:222:		WRITE_ONCE(iter->memcg, parent);
mm/memcontrol.c-223-	/* 3) Move already reparented objcgs to the parent's list */
--
mm/memcontrol.c=3823=mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
--
mm/memcontrol.c-3842-	if (parent) {
mm/memcontrol.c:3843:		WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent));
mm/memcontrol.c-3844-
--
mm/memcontrol.c-3848-		memcg->memory.track_failcnt = !memcg_on_dfl;
mm/memcontrol.c:3849:		WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable));
mm/memcontrol.c-3850-		page_counter_init(&memcg->kmem, &parent->kmem, false);
--
mm/page_counter.c=424=void page_counter_calculate_protection(struct page_counter *root,
--
mm/page_counter.c-452-
mm/page_counter.c:453:	WRITE_ONCE(counter->emin, effective_protection(usage, parent_usage,
mm/page_counter.c-454-			READ_ONCE(counter->min),
mm/page_counter.c:455:			READ_ONCE(parent->emin),
mm/page_counter.c-456-			atomic_long_read(&parent->children_min_usage),
--
mm/page_counter.c-458-
mm/page_counter.c:459:	WRITE_ONCE(counter->elow, effective_protection(usage, parent_usage,
mm/page_counter.c-460-			READ_ONCE(counter->low),
mm/page_counter.c:461:			READ_ONCE(parent->elow),
mm/page_counter.c-462-			atomic_long_read(&parent->children_low_usage),
--
tools/include/linux/rbtree_augmented.h=171=__rb_change_child(struct rb_node *old, struct rb_node *new,
--
tools/include/linux/rbtree_augmented.h-175-		if (parent->rb_left == old)
tools/include/linux/rbtree_augmented.h:176:			WRITE_ONCE(parent->rb_left, new);
tools/include/linux/rbtree_augmented.h-177-		else
tools/include/linux/rbtree_augmented.h:178:			WRITE_ONCE(parent->rb_right, new);
tools/include/linux/rbtree_augmented.h-179-	} else
--
tools/include/linux/rbtree_augmented.h=187=__rb_erase_augmented(struct rb_node *node, struct rb_root *root,
--
tools/include/linux/rbtree_augmented.h-257-			child2 = successor->rb_right;
tools/include/linux/rbtree_augmented.h:258:			WRITE_ONCE(parent->rb_left, child2);
tools/include/linux/rbtree_augmented.h-259-			WRITE_ONCE(successor->rb_right, child);
--
tools/lib/rbtree.c=85=__rb_insert(struct rb_node *node, struct rb_root *root,
--
tools/lib/rbtree.c-154-				tmp = node->rb_left;
tools/lib/rbtree.c:155:				WRITE_ONCE(parent->rb_right, tmp);
tools/lib/rbtree.c:156:				WRITE_ONCE(node->rb_left, parent);
tools/lib/rbtree.c-157-				if (tmp)
--
tools/lib/rbtree.c-175-			 */
tools/lib/rbtree.c:176:			WRITE_ONCE(gparent->rb_left, tmp); /* == parent->rb_right */
tools/lib/rbtree.c:177:			WRITE_ONCE(parent->rb_right, gparent);
tools/lib/rbtree.c-178-			if (tmp)
--
tools/lib/rbtree.c-198-				tmp = node->rb_right;
tools/lib/rbtree.c:199:				WRITE_ONCE(parent->rb_left, tmp);
tools/lib/rbtree.c:200:				WRITE_ONCE(node->rb_right, parent);
tools/lib/rbtree.c-201-				if (tmp)
--
tools/lib/rbtree.c-210-			/* Case 3 - left rotate at gparent */
tools/lib/rbtree.c:211:			WRITE_ONCE(gparent->rb_right, tmp); /* == parent->rb_left */
tools/lib/rbtree.c:212:			WRITE_ONCE(parent->rb_left, gparent);
tools/lib/rbtree.c-213-			if (tmp)
--
tools/lib/rbtree.c=227=____rb_erase_color(struct rb_node *parent, struct rb_root *root,
--
tools/lib/rbtree.c-252-				tmp1 = sibling->rb_left;
tools/lib/rbtree.c:253:				WRITE_ONCE(parent->rb_right, tmp1);
tools/lib/rbtree.c:254:				WRITE_ONCE(sibling->rb_left, parent);
tools/lib/rbtree.c-255-				rb_set_parent_color(tmp1, parent, RB_BLACK);
--
tools/lib/rbtree.c-321-				WRITE_ONCE(tmp2->rb_right, sibling);
tools/lib/rbtree.c:322:				WRITE_ONCE(parent->rb_right, tmp2);
tools/lib/rbtree.c-323-				if (tmp1)
--
tools/lib/rbtree.c-342-			tmp2 = sibling->rb_left;
tools/lib/rbtree.c:343:			WRITE_ONCE(parent->rb_right, tmp2);
tools/lib/rbtree.c:344:			WRITE_ONCE(sibling->rb_left, parent);
tools/lib/rbtree.c-345-			rb_set_parent_color(tmp1, sibling, RB_BLACK);
--
tools/lib/rbtree.c-356-				tmp1 = sibling->rb_right;
tools/lib/rbtree.c:357:				WRITE_ONCE(parent->rb_left, tmp1);
tools/lib/rbtree.c:358:				WRITE_ONCE(sibling->rb_right, parent);
tools/lib/rbtree.c-359-				rb_set_parent_color(tmp1, parent, RB_BLACK);
--
tools/lib/rbtree.c-385-				WRITE_ONCE(tmp2->rb_left, sibling);
tools/lib/rbtree.c:386:				WRITE_ONCE(parent->rb_left, tmp2);
tools/lib/rbtree.c-387-				if (tmp1)
--
tools/lib/rbtree.c-395-			tmp2 = sibling->rb_right;
tools/lib/rbtree.c:396:			WRITE_ONCE(parent->rb_left, tmp2);
tools/lib/rbtree.c:397:			WRITE_ONCE(sibling->rb_right, parent);
tools/lib/rbtree.c-398-			rb_set_parent_color(tmp1, sibling, RB_BLACK);
]

gemini-3-flash-preview

							input: 46652
							output: 18
							thoughts: 79
						

Okay, let's see...

**Checking for Concurrent Access Issues in the Maple Tree**

Alright, still no sign of `lib/maple_tree.c`. That's a bit perplexing. But before I start digging into the filesystem again, I should refocus. The immediate concern is concurrent access, specifically around that `parent` pointer.

My thinking is, before I start down the path of searching the entire codebase, let's focus. I need to verify – and *absolutely* confirm – whether `mte_set_node_dead` and `ma_dead_node` are the *only* functions where the `parent` pointer of a Maple Tree node is accessed in a potentially concurrent manner. A thorough review of those functions is a must-do before going any further.

Actually, that's not quite enough. Given the potential for subtle race conditions, I'll broaden my search within `lib/maple_tree.c`. I will be looking specifically for any usage of `READ_ONCE`. If `READ_ONCE` is used, it suggests that there may be concurrent access that needs attention, so that's the next check.

map[Expression:READ_ONCE]

map[Output:
Full output is too long, showing 500 out of 38374 lines.
Use more precise expression if possible.

[Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=78=lock-acquisition and lock-release functions::
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-85-    6   WRITE_ONCE(x, 1);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:86:    7   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-87-    8   raw_spin_unlock_rcu_node(rnp);
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-93-   14   WRITE_ONCE(y, 1);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:94:   15   r2 = READ_ONCE(z);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-95-   16   raw_spin_unlock_rcu_node(rnp);
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-101-   22   smp_mb();
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:102:   23   r3 = READ_ONCE(x);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-103-   24 }
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=112=The ``smp_mb__after_unlock_lock()`` invocations prevent this
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-133-|  g = get_state_synchronize_rcu()           smp_mb()                   |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:134:|  while (!poll_state_synchronize_rcu(g))    r1 = READ_ONCE(X)          |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-135-|          continue;                                                    |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:136:|  r0 = READ_ONCE(Y)                                                    |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-137-|                                                                       |
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=198=newly arrived RCU callbacks against future grace periods:
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-213-   12   /* Handle nohz enablement switches conservatively. */
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:214:   13   tne = READ_ONCE(tick_nohz_active);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-215-   14   if (tne != rdp->tick_nohz_enabled_snap) {
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-167-         x="255.60869"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:168:         y="309.29346">r1 = READ_ONCE(a);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-169-    <text
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-189-         x="396.10254"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:190:         y="384.71124">r2 = READ_ONCE(b);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-191-    <text
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-200-         x="396.10254"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:201:         y="582.13617">r3 = READ_ONCE(c);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-202-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-195-         x="255.60869"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:196:         y="297.29346">r1 = READ_ONCE(a);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-197-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-228-         x="396.10254"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:229:         y="572.13617">r2 = READ_ONCE(c);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-230-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-463-         x="541.70508"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:464:         y="387.6217">r3 = READ_ONCE(d);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-465-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-509-         x="686.27747"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:510:         y="461.83929">r4 = READ_ONCE(b);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-511-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-520-         x="686.27747"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:521:         y="669.26422">r5 = READ_ONCE(e);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-522-    <text
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=84=overhead to readers, for example:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-92-       5   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:93:       6   r1 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst:94:       7   r2 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-95-       8   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=138=recovery from node failure, more or less as follows:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-153-      12   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:154:      13   state_snap = READ_ONCE(state);
 Documentation/RCU/Design/Requirements/Requirements.rst-155-      14   if (state_snap == STATE_NORMAL)
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=467=resembling the dependency-ordering barrier that was later subsumed
 Documentation/RCU/Design/Requirements/Requirements.rst:468:into rcu_dereference() and later still into READ_ONCE(). The
 Documentation/RCU/Design/Requirements/Requirements.rst-469-need for these operations made itself known quite suddenly at a
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=702=threads:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-718-      13   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:719:      14   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-720-      15   rcu_read_unlock();
 Documentation/RCU/Design/Requirements/Requirements.rst-721-      16   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:722:      17   r2 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst-723-      18   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=737=significant ordering constraints would slow down these fast-path APIs.
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-745-+-----------------------------------------------------------------------+
 Documentation/RCU/Design/Requirements/Requirements.rst:746:| No, the volatile casts in READ_ONCE() and WRITE_ONCE()                |
 Documentation/RCU/Design/Requirements/Requirements.rst-747-| prevent the compiler from reordering in this particular case.         |
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=755=example illustrates this:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-761-       3   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:762:       4   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-763-       5   if (r1) {
 Documentation/RCU/Design/Requirements/Requirements.rst-764-       6     do_something_with_nonzero_x();
 Documentation/RCU/Design/Requirements/Requirements.rst:765:       7     r2 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst-766-       8     WARN_ON(!r2); /* BUG!!! */
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=819=are initially all zero:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-832-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:833:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-834-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-840-      18   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:841:      19   r2 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:842:      20   r3 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-843-      21   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=862=period is known to end before the second grace period starts:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-875-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:876:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-877-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-882-      17 {
 Documentation/RCU/Design/Requirements/Requirements.rst:883:      18   r2 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-884-      19   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-890-      25   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:891:      26   r3 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:892:      27   r4 = READ_ONCE(d);
 Documentation/RCU/Design/Requirements/Requirements.rst-893-      28   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=920=illustrated by the following, with all variables initially zero:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-933-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:934:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-935-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-942-      19   WRITE_ONCE(d, 1);
 Documentation/RCU/Design/Requirements/Requirements.rst:943:      20   r2 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-944-      21   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-948-      25 {
 Documentation/RCU/Design/Requirements/Requirements.rst:949:      26   r3 = READ_ONCE(d);
 Documentation/RCU/Design/Requirements/Requirements.rst-950-      27   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-956-      33   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:957:      34   r4 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:958:      35   r5 = READ_ONCE(e);
 Documentation/RCU/Design/Requirements/Requirements.rst-959-      36   rcu_read_unlock();
 --
 Documentation/RCU/checklist.rst=12=over a rather long period of time, but improvements are always welcome!
 --
 Documentation/RCU/checklist.rst-363-	time that readers might be accessing that structure.  In such
 Documentation/RCU/checklist.rst:364:	cases, READ_ONCE() may be used in place of rcu_dereference()
 Documentation/RCU/checklist.rst-365-	and the read-side markers (rcu_read_lock() and rcu_read_unlock(),
 --
 Documentation/RCU/listRCU.rst=145=has become list_for_each_entry_rcu().  The **_rcu()** list-traversal
 Documentation/RCU/listRCU.rst:146:primitives add READ_ONCE() and diagnostic checks for incorrect use
 Documentation/RCU/listRCU.rst-147-outside of an RCU read-side critical section.
 --
 Documentation/RCU/rcu_dereference.rst=25=readers working properly:
 --
 Documentation/RCU/rcu_dereference.rst-35-	return data preceding initialization that preceded the store
 Documentation/RCU/rcu_dereference.rst:36:	of the pointer.  (As noted later, in recent kernels READ_ONCE()
 Documentation/RCU/rcu_dereference.rst-37-	also prevents DEC Alpha from playing these tricks.)
 --
 Documentation/RCU/rcu_dereference.rst-45--	In the special case where data is added but is never removed
 Documentation/RCU/rcu_dereference.rst:46:	while readers are accessing the structure, READ_ONCE() may be used
 Documentation/RCU/rcu_dereference.rst:47:	instead of rcu_dereference().  In this case, use of READ_ONCE()
 Documentation/RCU/rcu_dereference.rst-48-	takes on the role of the lockless_dereference() primitive that
 --
 Documentation/RCU/whatisRCU.rst=687=don't forget about them when submitting patches making use of RCU!]::
 --
 Documentation/RCU/whatisRCU.rst-695-	({ \
 Documentation/RCU/whatisRCU.rst:696:		typeof(p) _________p1 = READ_ONCE(p); \
 Documentation/RCU/whatisRCU.rst-697-		(_________p1); \
 --
 Documentation/atomic_t.txt=82=The non-RMW ops are (typically) regular LOADs and STOREs and are canonically
 Documentation/atomic_t.txt:83:implemented using READ_ONCE(), WRITE_ONCE(), smp_load_acquire() and
 Documentation/atomic_t.txt-84-smp_store_release() respectively. Therefore, if you find yourself only using
 --
 Documentation/atomic_t.txt=119=with a lock:
 --
 Documentation/atomic_t.txt-124-    lock();
 Documentation/atomic_t.txt:125:    ret = READ_ONCE(v->counter); // == 1
 Documentation/atomic_t.txt-126-						atomic_set(v, 0);
 --
 Documentation/atomic_t.txt=234=strictly stronger than ACQUIRE. As illustrated:
 --
 Documentation/atomic_t.txt-242-  {
 Documentation/atomic_t.txt:243:    r0 = READ_ONCE(*x);
 Documentation/atomic_t.txt-244-    smp_rmb();
 --
 Documentation/core-api/circular-buffers.rst=154=The producer will look something like this::
 --
 Documentation/core-api/circular-buffers.rst-159-	/* The spin_unlock() and next spin_lock() provide needed ordering. */
 Documentation/core-api/circular-buffers.rst:160:	unsigned long tail = READ_ONCE(buffer->tail);
 Documentation/core-api/circular-buffers.rst-161-
 --
 Documentation/core-api/circular-buffers.rst=219=before it writes the new tail pointer, which will erase the item.
 Documentation/core-api/circular-buffers.rst-220-
 Documentation/core-api/circular-buffers.rst:221:Note the use of READ_ONCE() and smp_load_acquire() to read the
 Documentation/core-api/circular-buffers.rst-222-opposition index.  This prevents the compiler from discarding and
 --
 Documentation/core-api/errseq.rst=144=errseq_check_and_advance after taking the lock. e.g.::
 Documentation/core-api/errseq.rst-145-
 Documentation/core-api/errseq.rst:146:        if (errseq_check(&wd.wd_err, READ_ONCE(su.s_wd_err)) {
 Documentation/core-api/errseq.rst-147-                /* su.s_wd_err is protected by s_wd_err_lock */
 --
 Documentation/core-api/refcount-vs-atomic.rst=37=are executed in program order on a single CPU.
 Documentation/core-api/refcount-vs-atomic.rst:38:This is implemented using READ_ONCE()/WRITE_ONCE() and
 Documentation/core-api/refcount-vs-atomic.rst-39-compare-and-swap primitives.
 --
 Documentation/dev-tools/kcsan.rst=214=and if that code is free from data races.
 Documentation/dev-tools/kcsan.rst-215-
 Documentation/dev-tools/kcsan.rst:216:KCSAN is aware of *marked atomic operations* (``READ_ONCE``, ``WRITE_ONCE``,
 Documentation/dev-tools/kcsan.rst-217-``atomic_*``, etc.), and a subset of ordering guarantees implied by memory
 --
 Documentation/dev-tools/kcsan.rst=297=barrier. Consider the example::
 --
 Documentation/dev-tools/kcsan.rst-306-    {
 Documentation/dev-tools/kcsan.rst:307:        while (!READ_ONCE(flag));   // correct: smp_load_acquire(&flag)
 Documentation/dev-tools/kcsan.rst-308-        ... = x;                    // data race!
 --
 Documentation/driver-api/surface_aggregator/internal.rst=270=submission, i.e. cancellation, can not rely on the ``ptl`` reference to be
 Documentation/driver-api/surface_aggregator/internal.rst:271:set. Access to it in these functions is guarded by ``READ_ONCE()``, whereas
 Documentation/driver-api/surface_aggregator/internal.rst-272-setting ``ptl`` is equally guarded with ``WRITE_ONCE()`` for symmetry.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=275=them, specifically priority and state for tracing. In those cases, proper
 Documentation/driver-api/surface_aggregator/internal.rst:276:access is ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``. Such
 Documentation/driver-api/surface_aggregator/internal.rst-277-read-only access is only allowed when stale values are not critical.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=451=them, specifically the state for tracing. In those cases, proper access is
 Documentation/driver-api/surface_aggregator/internal.rst:452:ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``. Such read-only
 Documentation/driver-api/surface_aggregator/internal.rst-453-access is only allowed when stale values are not critical.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=572=invalid usages, but rather aim to help catch them. In those cases, proper
 Documentation/driver-api/surface_aggregator/internal.rst:573:variable access is ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``.
 Documentation/driver-api/surface_aggregator/internal.rst-574-
 --
 Documentation/filesystems/path-lookup.rst=896=similar.
 Documentation/filesystems/path-lookup.rst-897-
 Documentation/filesystems/path-lookup.rst:898:.. _READ_ONCE: https://lwn.net/Articles/624126/
 Documentation/filesystems/path-lookup.rst-899-
 --
 Documentation/filesystems/path-lookup.rst=904=when accessing fields in the dentry.  This "extra care" typically
 Documentation/filesystems/path-lookup.rst:905:involves using  `READ_ONCE() <READ_ONCE_>`_ to access fields, and verifying the
 Documentation/filesystems/path-lookup.rst-906-result is not NULL before using it.  This pattern can be seen in
 --
 Documentation/gpu/todo.rst=932=struct drm_sched_rq is read at many places without any locks, not even with a
 Documentation/gpu/todo.rst:933:READ_ONCE. At XDC 2025 no one could really tell why that is the case, whether
 Documentation/gpu/todo.rst-934-locks are needed and whether they could be added. (But for real, that should
 --
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus=14=P0(int *x, atomic_t *y)
 --
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus-18-
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus:19:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus-20-	smp_rmb();
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus=12=P0(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-19-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus:20:	r0 = READ_ONCE(*y);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-21-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus=23=P1(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-30-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus:31:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-32-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus=20=P1(int *x, int *y)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus-26-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus:27:	r2 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus-28-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus=13=P0(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-19-	r1 = cmpxchg(z, 1, 0);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus:20:	r0 = READ_ONCE(*y);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-21-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus=23=P1(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-29-	r1 = cmpxchg(z, 1, 0);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus:30:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-31-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus=21=P1(int *x, int *y)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus-26-	r1 = cmpxchg(y, 0, 1);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus:27:	r2 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus-28-}
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus=15=P0(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-20-
 Documentation/litmus-tests/locking/DCL-broken.litmus:21:	r0 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-22-	if (r0 == 0) {
 Documentation/litmus-tests/locking/DCL-broken.litmus-23-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-broken.litmus:24:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-25-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-30-	}
 Documentation/litmus-tests/locking/DCL-broken.litmus:31:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-broken.litmus-32-}
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus=34=P1(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-39-
 Documentation/litmus-tests/locking/DCL-broken.litmus:40:	r0 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-41-	if (r0 == 0) {
 Documentation/litmus-tests/locking/DCL-broken.litmus-42-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-broken.litmus:43:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-44-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-49-	}
 Documentation/litmus-tests/locking/DCL-broken.litmus:50:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-broken.litmus-51-}
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus=16=P0(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-24-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-fixed.litmus:25:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-26-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-31-	}
 Documentation/litmus-tests/locking/DCL-fixed.litmus:32:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-33-}
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus=35=P1(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-43-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-fixed.litmus:44:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-45-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-50-	}
 Documentation/litmus-tests/locking/DCL-fixed.litmus:51:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-52-}
 --
 Documentation/litmus-tests/locking/RM-broken.litmus=26=P1(int *x, atomic_t *y, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/RM-broken.litmus-32-	spin_lock(lck);
 Documentation/litmus-tests/locking/RM-broken.litmus:33:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-broken.litmus:34:	r1 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-broken.litmus-35-	r2 = atomic_inc_return(y);
 --
 Documentation/litmus-tests/locking/RM-fixed.litmus=26=P1(int *x, atomic_t *y, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/RM-fixed.litmus-31-
 Documentation/litmus-tests/locking/RM-fixed.litmus:32:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-fixed.litmus:33:	r1 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-fixed.litmus-34-	spin_lock(lck);
 --
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus=24=P0(int *x, int *z, int **y)
 --
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus-30-	r0 = rcu_dereference(*y);
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus:31:	r1 = READ_ONCE(*r0);
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus-32-	rcu_read_unlock();
 --
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus=27=P1(int *x, int *y)
 --
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-31-
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus:32:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-33-	synchronize_rcu();
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus:34:	r1 = READ_ONCE(*y);
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-35-}
 --
 Documentation/livepatch/callbacks.rst=119=so that tcp_send_challenge_ack() could first be changed to read
 Documentation/livepatch/callbacks.rst:120:sysctl_tcp_challenge_ack_limit with READ_ONCE.
 Documentation/livepatch/callbacks.rst-121-
 --
 Documentation/memory-barriers.txt=226=There are some minimal guarantees that may be expected of a CPU:
 --
 Documentation/memory-barriers.txt-230-
 Documentation/memory-barriers.txt:231:	Q = READ_ONCE(P); D = READ_ONCE(*Q);
 Documentation/memory-barriers.txt-232-
 --
 Documentation/memory-barriers.txt-236-
 Documentation/memory-barriers.txt:237:     and always in that order.  However, on DEC Alpha, READ_ONCE() also
 Documentation/memory-barriers.txt-238-     emits a memory-barrier instruction, so that a DEC Alpha CPU will
 --
 Documentation/memory-barriers.txt-242-
 Documentation/memory-barriers.txt:243:     Whether on DEC Alpha or not, the READ_ONCE() also prevents compiler
 Documentation/memory-barriers.txt-244-     mischief.
 --
 Documentation/memory-barriers.txt-248-
 Documentation/memory-barriers.txt:249:	a = READ_ONCE(*X); WRITE_ONCE(*X, b);
 Documentation/memory-barriers.txt-250-
 --
 Documentation/memory-barriers.txt-256-
 Documentation/memory-barriers.txt:257:	WRITE_ONCE(*X, c); d = READ_ONCE(*X);
 Documentation/memory-barriers.txt-258-
 --
 Documentation/memory-barriers.txt=266=And there are a number of things that _must_ or _must_not_ be assumed:
 --
 Documentation/memory-barriers.txt-268- (*) It _must_not_ be assumed that the compiler will do what you want
 Documentation/memory-barriers.txt:269:     with memory references that are not protected by READ_ONCE() and
 Documentation/memory-barriers.txt-270-     WRITE_ONCE().  Without them, the compiler is within its rights to
 --
 Documentation/memory-barriers.txt=377=Memory barriers come in four basic varieties:
 --
 Documentation/memory-barriers.txt-438-     dependency barriers.  Nowadays, APIs for marking loads from shared
 Documentation/memory-barriers.txt:439:     variables such as READ_ONCE() and rcu_dereference() provide implicit
 Documentation/memory-barriers.txt-440-     address-dependency barriers.
 --
 Documentation/memory-barriers.txt=567=see Documentation/RCU/rcu_dereference.rst.
 Documentation/memory-barriers.txt-568-
 Documentation/memory-barriers.txt:569:As of v4.15 of the Linux kernel, an smp_mb() was added to READ_ONCE() for
 Documentation/memory-barriers.txt-570-DEC Alpha, which means that about the only people who need to pay attention
 Documentation/memory-barriers.txt=571=to this section are those working on DEC Alpha architecture-specific code
 Documentation/memory-barriers.txt:572:and those working on READ_ONCE() itself.  For those who need it, and for
 Documentation/memory-barriers.txt-573-those who are interested in the history, here is the story of
 --
 Documentation/memory-barriers.txt=582=following sequence of events:
 --
 Documentation/memory-barriers.txt-589-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:590:			      Q = READ_ONCE_OLD(P);
 Documentation/memory-barriers.txt-591-			      D = *Q;
 Documentation/memory-barriers.txt-592-
 Documentation/memory-barriers.txt:593:[!] READ_ONCE_OLD() corresponds to READ_ONCE() of pre-4.15 kernel, which
 Documentation/memory-barriers.txt-594-doesn't imply an address-dependency barrier.
 --
 Documentation/memory-barriers.txt=609=Alpha).
 Documentation/memory-barriers.txt-610-
 Documentation/memory-barriers.txt:611:To deal with this, READ_ONCE() provides an implicit address-dependency barrier
 Documentation/memory-barriers.txt-612-since kernel release v4.15:
 --
 Documentation/memory-barriers.txt-619-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:620:			      Q = READ_ONCE(P);
 Documentation/memory-barriers.txt-621-			      <implicit address-dependency barrier>
 --
 Documentation/memory-barriers.txt=644=dependencies in a great many highly creative ways.
 --
 Documentation/memory-barriers.txt-651-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:652:			      Q = READ_ONCE_OLD(P);
 Documentation/memory-barriers.txt-653-			      WRITE_ONCE(*Q, 5);
 --
 Documentation/memory-barriers.txt=656=Q with the store into *Q.  In other words, this outcome is prohibited,
 Documentation/memory-barriers.txt:657:even without an implicit address-dependency barrier of modern READ_ONCE():
 Documentation/memory-barriers.txt-658-
 --
 Documentation/memory-barriers.txt=689=Consider the following bit of code:
 Documentation/memory-barriers.txt-690-
 Documentation/memory-barriers.txt:691:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-692-	<implicit address-dependency barrier>
 --
 Documentation/memory-barriers.txt-694-		/* BUG: No address dependency!!! */
 Documentation/memory-barriers.txt:695:		p = READ_ONCE(b);
 Documentation/memory-barriers.txt-696-	}
 --
 Documentation/memory-barriers.txt=702=what's actually required is:
 Documentation/memory-barriers.txt-703-
 Documentation/memory-barriers.txt:704:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-705-	if (q) {
 Documentation/memory-barriers.txt-706-		<read barrier>
 Documentation/memory-barriers.txt:707:		p = READ_ONCE(b);
 Documentation/memory-barriers.txt-708-	}
 --
 Documentation/memory-barriers.txt=711=for load-store control dependencies, as in the following example:
 Documentation/memory-barriers.txt-712-
 Documentation/memory-barriers.txt:713:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-714-	if (q) {
 --
 Documentation/memory-barriers.txt=718=Control dependencies pair normally with other types of barriers.
 Documentation/memory-barriers.txt:719:That said, please note that neither READ_ONCE() nor WRITE_ONCE()
 Documentation/memory-barriers.txt:720:are optional! Without the READ_ONCE(), the compiler might combine the
 Documentation/memory-barriers.txt-721-load from 'a' with other loads from 'a'.  Without the WRITE_ONCE(),
 --
 Documentation/memory-barriers.txt=728=as follows:
 --
 Documentation/memory-barriers.txt-732-
 Documentation/memory-barriers.txt:733:So don't leave out the READ_ONCE().
 Documentation/memory-barriers.txt-734-
 --
 Documentation/memory-barriers.txt=736=branches of the "if" statement as follows:
 Documentation/memory-barriers.txt-737-
 Documentation/memory-barriers.txt:738:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-739-	if (q) {
 --
 Documentation/memory-barriers.txt=750=optimization levels:
 Documentation/memory-barriers.txt-751-
 Documentation/memory-barriers.txt:752:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-753-	barrier();
 --
 Documentation/memory-barriers.txt=768=memory barriers, for example, smp_store_release():
 Documentation/memory-barriers.txt-769-
 Documentation/memory-barriers.txt:770:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-771-	if (q) {
 --
 Documentation/memory-barriers.txt=780=ordering is guaranteed only when the stores differ, for example:
 Documentation/memory-barriers.txt-781-
 Documentation/memory-barriers.txt:782:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-783-	if (q) {
 --
 Documentation/memory-barriers.txt-790-
 Documentation/memory-barriers.txt:791:The initial READ_ONCE() is still required to prevent the compiler from
 Documentation/memory-barriers.txt-792-proving the value of 'a'.
 --
 Documentation/memory-barriers.txt=796=the needed conditional.  For example:
 Documentation/memory-barriers.txt-797-
 Documentation/memory-barriers.txt:798:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-799-	if (q % MAX) {
]
]

gemini-3-flash-preview

							input: 58118
							output: 27
							thoughts: 25
						

Here's my analysis of the situation, presented from my perspective as an expert:

**Diving into `lib/maple_tree.c` and `READ_ONCE`**

Alright, the objective is pretty clear. I need to get a better handle on how `READ_ONCE` is being employed within the context of the Maple Tree implementation, specifically within `lib/maple_tree.c`. My immediate thought is, I need to understand its specific usage in this file to provide meaningful information or analysis. I'll need to carefully examine the surrounding code to grasp the intent behind the `READ_ONCE` calls. Are we dealing with volatile data, shared memory scenarios, or something else entirely? What aspects of concurrency are they guarding against here? I know this likely isn't a simple thing. It will be important to quickly understand which data structures are using this approach, and how it relates to other related mechanisms used for memory access and synchronization within the Maple Tree. My experience tells me there will be specific use cases, and the devil will be in the details. Time to get my hands dirty.

no matches

map[Expression:READ_ONCE.*lib/maple_tree.c]

map[Output:]

gemini-3-flash-preview

							input: 58417
							output: 27
							thoughts: 24
						

Here's my attempt at summarizing the thought process, tailored for an expert audience:

**Investigating `READ_ONCE` in `lib/maple_tree.c`**

Okay, I need to understand how `READ_ONCE` is used in `lib/maple_tree.c`. It's a key primitive for memory ordering, and I'm specifically interested in its interactions within the Maple Tree implementation. Rather than blindly staring at the code, I'll go directly for the occurrences of `READ_ONCE`. A quick `grep` will be far more efficient to get a handle on the direct usage. This should quickly highlight critical read operations that need special consideration for concurrency. Then I can dig into the context of those lines, evaluate what ordering is used on the memory, and evaluate the lock mechanisms around the specific accesses.

no matches

map[Expression:READ_ONCE.*lib/maple_tree.c]

map[Output:]

gemini-3-flash-preview

							input: 58660
							output: 18
							thoughts: 30
						

Here's my attempt at summarizing the thought process, as if I were thinking it:

**Rethinking My Approach**

Hold on a second... I'm going about this all wrong. I'm getting bogged down in the overall strategy, and I need to be more focused. Instead of thinking so broadly, I need to pinpoint the exact file I'm interested in. Forget the general approach for now. I should be searching specifically within `lib/maple_tree.c`. That's the key starting point for now. Focusing my search scope will give me immediate, focused results.

map[Expression:READ_ONCE]

map[Output:
Full output is too long, showing 500 out of 38374 lines.
Use more precise expression if possible.

[Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=78=lock-acquisition and lock-release functions::
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-85-    6   WRITE_ONCE(x, 1);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:86:    7   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-87-    8   raw_spin_unlock_rcu_node(rnp);
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-93-   14   WRITE_ONCE(y, 1);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:94:   15   r2 = READ_ONCE(z);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-95-   16   raw_spin_unlock_rcu_node(rnp);
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-101-   22   smp_mb();
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:102:   23   r3 = READ_ONCE(x);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-103-   24 }
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=112=The ``smp_mb__after_unlock_lock()`` invocations prevent this
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-133-|  g = get_state_synchronize_rcu()           smp_mb()                   |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:134:|  while (!poll_state_synchronize_rcu(g))    r1 = READ_ONCE(X)          |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-135-|          continue;                                                    |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:136:|  r0 = READ_ONCE(Y)                                                    |
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-137-|                                                                       |
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst=198=newly arrived RCU callbacks against future grace periods:
 --
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-213-   12   /* Handle nohz enablement switches conservatively. */
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst:214:   13   tne = READ_ONCE(tick_nohz_active);
 Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst-215-   14   if (tne != rdp->tick_nohz_enabled_snap) {
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-167-         x="255.60869"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:168:         y="309.29346">r1 = READ_ONCE(a);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-169-    <text
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-189-         x="396.10254"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:190:         y="384.71124">r2 = READ_ONCE(b);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-191-    <text
 --
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-200-         x="396.10254"
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg:201:         y="582.13617">r3 = READ_ONCE(c);</tspan></text>
 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg-202-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-195-         x="255.60869"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:196:         y="297.29346">r1 = READ_ONCE(a);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-197-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-228-         x="396.10254"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:229:         y="572.13617">r2 = READ_ONCE(c);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-230-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-463-         x="541.70508"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:464:         y="387.6217">r3 = READ_ONCE(d);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-465-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-509-         x="686.27747"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:510:         y="461.83929">r4 = READ_ONCE(b);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-511-    <text
 --
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-520-         x="686.27747"
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg:521:         y="669.26422">r5 = READ_ONCE(e);</tspan></text>
 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg-522-    <text
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=84=overhead to readers, for example:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-92-       5   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:93:       6   r1 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst:94:       7   r2 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-95-       8   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=138=recovery from node failure, more or less as follows:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-153-      12   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:154:      13   state_snap = READ_ONCE(state);
 Documentation/RCU/Design/Requirements/Requirements.rst-155-      14   if (state_snap == STATE_NORMAL)
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=467=resembling the dependency-ordering barrier that was later subsumed
 Documentation/RCU/Design/Requirements/Requirements.rst:468:into rcu_dereference() and later still into READ_ONCE(). The
 Documentation/RCU/Design/Requirements/Requirements.rst-469-need for these operations made itself known quite suddenly at a
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=702=threads:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-718-      13   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:719:      14   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-720-      15   rcu_read_unlock();
 Documentation/RCU/Design/Requirements/Requirements.rst-721-      16   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:722:      17   r2 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst-723-      18   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=737=significant ordering constraints would slow down these fast-path APIs.
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-745-+-----------------------------------------------------------------------+
 Documentation/RCU/Design/Requirements/Requirements.rst:746:| No, the volatile casts in READ_ONCE() and WRITE_ONCE()                |
 Documentation/RCU/Design/Requirements/Requirements.rst-747-| prevent the compiler from reordering in this particular case.         |
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=755=example illustrates this:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-761-       3   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:762:       4   r1 = READ_ONCE(y);
 Documentation/RCU/Design/Requirements/Requirements.rst-763-       5   if (r1) {
 Documentation/RCU/Design/Requirements/Requirements.rst-764-       6     do_something_with_nonzero_x();
 Documentation/RCU/Design/Requirements/Requirements.rst:765:       7     r2 = READ_ONCE(x);
 Documentation/RCU/Design/Requirements/Requirements.rst-766-       8     WARN_ON(!r2); /* BUG!!! */
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=819=are initially all zero:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-832-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:833:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-834-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-840-      18   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:841:      19   r2 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:842:      20   r3 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-843-      21   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=862=period is known to end before the second grace period starts:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-875-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:876:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-877-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-882-      17 {
 Documentation/RCU/Design/Requirements/Requirements.rst:883:      18   r2 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-884-      19   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-890-      25   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:891:      26   r3 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:892:      27   r4 = READ_ONCE(d);
 Documentation/RCU/Design/Requirements/Requirements.rst-893-      28   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst=920=illustrated by the following, with all variables initially zero:
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-933-      10 {
 Documentation/RCU/Design/Requirements/Requirements.rst:934:      11   r1 = READ_ONCE(a);
 Documentation/RCU/Design/Requirements/Requirements.rst-935-      12   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-942-      19   WRITE_ONCE(d, 1);
 Documentation/RCU/Design/Requirements/Requirements.rst:943:      20   r2 = READ_ONCE(c);
 Documentation/RCU/Design/Requirements/Requirements.rst-944-      21   rcu_read_unlock();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-948-      25 {
 Documentation/RCU/Design/Requirements/Requirements.rst:949:      26   r3 = READ_ONCE(d);
 Documentation/RCU/Design/Requirements/Requirements.rst-950-      27   synchronize_rcu();
 --
 Documentation/RCU/Design/Requirements/Requirements.rst-956-      33   rcu_read_lock();
 Documentation/RCU/Design/Requirements/Requirements.rst:957:      34   r4 = READ_ONCE(b);
 Documentation/RCU/Design/Requirements/Requirements.rst:958:      35   r5 = READ_ONCE(e);
 Documentation/RCU/Design/Requirements/Requirements.rst-959-      36   rcu_read_unlock();
 --
 Documentation/RCU/checklist.rst=12=over a rather long period of time, but improvements are always welcome!
 --
 Documentation/RCU/checklist.rst-363-	time that readers might be accessing that structure.  In such
 Documentation/RCU/checklist.rst:364:	cases, READ_ONCE() may be used in place of rcu_dereference()
 Documentation/RCU/checklist.rst-365-	and the read-side markers (rcu_read_lock() and rcu_read_unlock(),
 --
 Documentation/RCU/listRCU.rst=145=has become list_for_each_entry_rcu().  The **_rcu()** list-traversal
 Documentation/RCU/listRCU.rst:146:primitives add READ_ONCE() and diagnostic checks for incorrect use
 Documentation/RCU/listRCU.rst-147-outside of an RCU read-side critical section.
 --
 Documentation/RCU/rcu_dereference.rst=25=readers working properly:
 --
 Documentation/RCU/rcu_dereference.rst-35-	return data preceding initialization that preceded the store
 Documentation/RCU/rcu_dereference.rst:36:	of the pointer.  (As noted later, in recent kernels READ_ONCE()
 Documentation/RCU/rcu_dereference.rst-37-	also prevents DEC Alpha from playing these tricks.)
 --
 Documentation/RCU/rcu_dereference.rst-45--	In the special case where data is added but is never removed
 Documentation/RCU/rcu_dereference.rst:46:	while readers are accessing the structure, READ_ONCE() may be used
 Documentation/RCU/rcu_dereference.rst:47:	instead of rcu_dereference().  In this case, use of READ_ONCE()
 Documentation/RCU/rcu_dereference.rst-48-	takes on the role of the lockless_dereference() primitive that
 --
 Documentation/RCU/whatisRCU.rst=687=don't forget about them when submitting patches making use of RCU!]::
 --
 Documentation/RCU/whatisRCU.rst-695-	({ \
 Documentation/RCU/whatisRCU.rst:696:		typeof(p) _________p1 = READ_ONCE(p); \
 Documentation/RCU/whatisRCU.rst-697-		(_________p1); \
 --
 Documentation/atomic_t.txt=82=The non-RMW ops are (typically) regular LOADs and STOREs and are canonically
 Documentation/atomic_t.txt:83:implemented using READ_ONCE(), WRITE_ONCE(), smp_load_acquire() and
 Documentation/atomic_t.txt-84-smp_store_release() respectively. Therefore, if you find yourself only using
 --
 Documentation/atomic_t.txt=119=with a lock:
 --
 Documentation/atomic_t.txt-124-    lock();
 Documentation/atomic_t.txt:125:    ret = READ_ONCE(v->counter); // == 1
 Documentation/atomic_t.txt-126-						atomic_set(v, 0);
 --
 Documentation/atomic_t.txt=234=strictly stronger than ACQUIRE. As illustrated:
 --
 Documentation/atomic_t.txt-242-  {
 Documentation/atomic_t.txt:243:    r0 = READ_ONCE(*x);
 Documentation/atomic_t.txt-244-    smp_rmb();
 --
 Documentation/core-api/circular-buffers.rst=154=The producer will look something like this::
 --
 Documentation/core-api/circular-buffers.rst-159-	/* The spin_unlock() and next spin_lock() provide needed ordering. */
 Documentation/core-api/circular-buffers.rst:160:	unsigned long tail = READ_ONCE(buffer->tail);
 Documentation/core-api/circular-buffers.rst-161-
 --
 Documentation/core-api/circular-buffers.rst=219=before it writes the new tail pointer, which will erase the item.
 Documentation/core-api/circular-buffers.rst-220-
 Documentation/core-api/circular-buffers.rst:221:Note the use of READ_ONCE() and smp_load_acquire() to read the
 Documentation/core-api/circular-buffers.rst-222-opposition index.  This prevents the compiler from discarding and
 --
 Documentation/core-api/errseq.rst=144=errseq_check_and_advance after taking the lock. e.g.::
 Documentation/core-api/errseq.rst-145-
 Documentation/core-api/errseq.rst:146:        if (errseq_check(&wd.wd_err, READ_ONCE(su.s_wd_err)) {
 Documentation/core-api/errseq.rst-147-                /* su.s_wd_err is protected by s_wd_err_lock */
 --
 Documentation/core-api/refcount-vs-atomic.rst=37=are executed in program order on a single CPU.
 Documentation/core-api/refcount-vs-atomic.rst:38:This is implemented using READ_ONCE()/WRITE_ONCE() and
 Documentation/core-api/refcount-vs-atomic.rst-39-compare-and-swap primitives.
 --
 Documentation/dev-tools/kcsan.rst=214=and if that code is free from data races.
 Documentation/dev-tools/kcsan.rst-215-
 Documentation/dev-tools/kcsan.rst:216:KCSAN is aware of *marked atomic operations* (``READ_ONCE``, ``WRITE_ONCE``,
 Documentation/dev-tools/kcsan.rst-217-``atomic_*``, etc.), and a subset of ordering guarantees implied by memory
 --
 Documentation/dev-tools/kcsan.rst=297=barrier. Consider the example::
 --
 Documentation/dev-tools/kcsan.rst-306-    {
 Documentation/dev-tools/kcsan.rst:307:        while (!READ_ONCE(flag));   // correct: smp_load_acquire(&flag)
 Documentation/dev-tools/kcsan.rst-308-        ... = x;                    // data race!
 --
 Documentation/driver-api/surface_aggregator/internal.rst=270=submission, i.e. cancellation, can not rely on the ``ptl`` reference to be
 Documentation/driver-api/surface_aggregator/internal.rst:271:set. Access to it in these functions is guarded by ``READ_ONCE()``, whereas
 Documentation/driver-api/surface_aggregator/internal.rst-272-setting ``ptl`` is equally guarded with ``WRITE_ONCE()`` for symmetry.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=275=them, specifically priority and state for tracing. In those cases, proper
 Documentation/driver-api/surface_aggregator/internal.rst:276:access is ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``. Such
 Documentation/driver-api/surface_aggregator/internal.rst-277-read-only access is only allowed when stale values are not critical.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=451=them, specifically the state for tracing. In those cases, proper access is
 Documentation/driver-api/surface_aggregator/internal.rst:452:ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``. Such read-only
 Documentation/driver-api/surface_aggregator/internal.rst-453-access is only allowed when stale values are not critical.
 --
 Documentation/driver-api/surface_aggregator/internal.rst=572=invalid usages, but rather aim to help catch them. In those cases, proper
 Documentation/driver-api/surface_aggregator/internal.rst:573:variable access is ensured by employing ``WRITE_ONCE()`` and ``READ_ONCE()``.
 Documentation/driver-api/surface_aggregator/internal.rst-574-
 --
 Documentation/filesystems/path-lookup.rst=896=similar.
 Documentation/filesystems/path-lookup.rst-897-
 Documentation/filesystems/path-lookup.rst:898:.. _READ_ONCE: https://lwn.net/Articles/624126/
 Documentation/filesystems/path-lookup.rst-899-
 --
 Documentation/filesystems/path-lookup.rst=904=when accessing fields in the dentry.  This "extra care" typically
 Documentation/filesystems/path-lookup.rst:905:involves using  `READ_ONCE() <READ_ONCE_>`_ to access fields, and verifying the
 Documentation/filesystems/path-lookup.rst-906-result is not NULL before using it.  This pattern can be seen in
 --
 Documentation/gpu/todo.rst=932=struct drm_sched_rq is read at many places without any locks, not even with a
 Documentation/gpu/todo.rst:933:READ_ONCE. At XDC 2025 no one could really tell why that is the case, whether
 Documentation/gpu/todo.rst-934-locks are needed and whether they could be added. (But for real, that should
 --
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus=14=P0(int *x, atomic_t *y)
 --
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus-18-
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus:19:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/Atomic-RMW+mb__after_atomic-is-stronger-than-acquire.litmus-20-	smp_rmb();
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus=12=P0(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-19-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus:20:	r0 = READ_ONCE(*y);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-21-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus=23=P1(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-30-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus:31:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-1.litmus-32-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus=20=P1(int *x, int *y)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus-26-	smp_mb__after_atomic();
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus:27:	r2 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-ordered-2.litmus-28-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus=13=P0(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-19-	r1 = cmpxchg(z, 1, 0);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus:20:	r0 = READ_ONCE(*y);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-21-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus=23=P1(int *x, int *y, int *z)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-29-	r1 = cmpxchg(z, 1, 0);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus:30:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-1.litmus-31-}
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus=21=P1(int *x, int *y)
 --
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus-26-	r1 = cmpxchg(y, 0, 1);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus:27:	r2 = READ_ONCE(*x);
 Documentation/litmus-tests/atomic/cmpxchg-fail-unordered-2.litmus-28-}
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus=15=P0(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-20-
 Documentation/litmus-tests/locking/DCL-broken.litmus:21:	r0 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-22-	if (r0 == 0) {
 Documentation/litmus-tests/locking/DCL-broken.litmus-23-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-broken.litmus:24:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-25-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-30-	}
 Documentation/litmus-tests/locking/DCL-broken.litmus:31:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-broken.litmus-32-}
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus=34=P1(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-39-
 Documentation/litmus-tests/locking/DCL-broken.litmus:40:	r0 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-41-	if (r0 == 0) {
 Documentation/litmus-tests/locking/DCL-broken.litmus-42-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-broken.litmus:43:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-broken.litmus-44-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-broken.litmus-49-	}
 Documentation/litmus-tests/locking/DCL-broken.litmus:50:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-broken.litmus-51-}
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus=16=P0(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-24-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-fixed.litmus:25:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-26-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-31-	}
 Documentation/litmus-tests/locking/DCL-fixed.litmus:32:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-33-}
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus=35=P1(int *flag, int *data, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-43-		spin_lock(lck);
 Documentation/litmus-tests/locking/DCL-fixed.litmus:44:		r1 = READ_ONCE(*flag);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-45-		if (r1 == 0) {
 --
 Documentation/litmus-tests/locking/DCL-fixed.litmus-50-	}
 Documentation/litmus-tests/locking/DCL-fixed.litmus:51:	r2 = READ_ONCE(*data);
 Documentation/litmus-tests/locking/DCL-fixed.litmus-52-}
 --
 Documentation/litmus-tests/locking/RM-broken.litmus=26=P1(int *x, atomic_t *y, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/RM-broken.litmus-32-	spin_lock(lck);
 Documentation/litmus-tests/locking/RM-broken.litmus:33:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-broken.litmus:34:	r1 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-broken.litmus-35-	r2 = atomic_inc_return(y);
 --
 Documentation/litmus-tests/locking/RM-fixed.litmus=26=P1(int *x, atomic_t *y, spinlock_t *lck)
 --
 Documentation/litmus-tests/locking/RM-fixed.litmus-31-
 Documentation/litmus-tests/locking/RM-fixed.litmus:32:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-fixed.litmus:33:	r1 = READ_ONCE(*x);
 Documentation/litmus-tests/locking/RM-fixed.litmus-34-	spin_lock(lck);
 --
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus=24=P0(int *x, int *z, int **y)
 --
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus-30-	r0 = rcu_dereference(*y);
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus:31:	r1 = READ_ONCE(*r0);
 Documentation/litmus-tests/rcu/RCU+sync+free.litmus-32-	rcu_read_unlock();
 --
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus=27=P1(int *x, int *y)
 --
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-31-
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus:32:	r0 = READ_ONCE(*x);
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-33-	synchronize_rcu();
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus:34:	r1 = READ_ONCE(*y);
 Documentation/litmus-tests/rcu/RCU+sync+read.litmus-35-}
 --
 Documentation/livepatch/callbacks.rst=119=so that tcp_send_challenge_ack() could first be changed to read
 Documentation/livepatch/callbacks.rst:120:sysctl_tcp_challenge_ack_limit with READ_ONCE.
 Documentation/livepatch/callbacks.rst-121-
 --
 Documentation/memory-barriers.txt=226=There are some minimal guarantees that may be expected of a CPU:
 --
 Documentation/memory-barriers.txt-230-
 Documentation/memory-barriers.txt:231:	Q = READ_ONCE(P); D = READ_ONCE(*Q);
 Documentation/memory-barriers.txt-232-
 --
 Documentation/memory-barriers.txt-236-
 Documentation/memory-barriers.txt:237:     and always in that order.  However, on DEC Alpha, READ_ONCE() also
 Documentation/memory-barriers.txt-238-     emits a memory-barrier instruction, so that a DEC Alpha CPU will
 --
 Documentation/memory-barriers.txt-242-
 Documentation/memory-barriers.txt:243:     Whether on DEC Alpha or not, the READ_ONCE() also prevents compiler
 Documentation/memory-barriers.txt-244-     mischief.
 --
 Documentation/memory-barriers.txt-248-
 Documentation/memory-barriers.txt:249:	a = READ_ONCE(*X); WRITE_ONCE(*X, b);
 Documentation/memory-barriers.txt-250-
 --
 Documentation/memory-barriers.txt-256-
 Documentation/memory-barriers.txt:257:	WRITE_ONCE(*X, c); d = READ_ONCE(*X);
 Documentation/memory-barriers.txt-258-
 --
 Documentation/memory-barriers.txt=266=And there are a number of things that _must_ or _must_not_ be assumed:
 --
 Documentation/memory-barriers.txt-268- (*) It _must_not_ be assumed that the compiler will do what you want
 Documentation/memory-barriers.txt:269:     with memory references that are not protected by READ_ONCE() and
 Documentation/memory-barriers.txt-270-     WRITE_ONCE().  Without them, the compiler is within its rights to
 --
 Documentation/memory-barriers.txt=377=Memory barriers come in four basic varieties:
 --
 Documentation/memory-barriers.txt-438-     dependency barriers.  Nowadays, APIs for marking loads from shared
 Documentation/memory-barriers.txt:439:     variables such as READ_ONCE() and rcu_dereference() provide implicit
 Documentation/memory-barriers.txt-440-     address-dependency barriers.
 --
 Documentation/memory-barriers.txt=567=see Documentation/RCU/rcu_dereference.rst.
 Documentation/memory-barriers.txt-568-
 Documentation/memory-barriers.txt:569:As of v4.15 of the Linux kernel, an smp_mb() was added to READ_ONCE() for
 Documentation/memory-barriers.txt-570-DEC Alpha, which means that about the only people who need to pay attention
 Documentation/memory-barriers.txt=571=to this section are those working on DEC Alpha architecture-specific code
 Documentation/memory-barriers.txt:572:and those working on READ_ONCE() itself.  For those who need it, and for
 Documentation/memory-barriers.txt-573-those who are interested in the history, here is the story of
 --
 Documentation/memory-barriers.txt=582=following sequence of events:
 --
 Documentation/memory-barriers.txt-589-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:590:			      Q = READ_ONCE_OLD(P);
 Documentation/memory-barriers.txt-591-			      D = *Q;
 Documentation/memory-barriers.txt-592-
 Documentation/memory-barriers.txt:593:[!] READ_ONCE_OLD() corresponds to READ_ONCE() of pre-4.15 kernel, which
 Documentation/memory-barriers.txt-594-doesn't imply an address-dependency barrier.
 --
 Documentation/memory-barriers.txt=609=Alpha).
 Documentation/memory-barriers.txt-610-
 Documentation/memory-barriers.txt:611:To deal with this, READ_ONCE() provides an implicit address-dependency barrier
 Documentation/memory-barriers.txt-612-since kernel release v4.15:
 --
 Documentation/memory-barriers.txt-619-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:620:			      Q = READ_ONCE(P);
 Documentation/memory-barriers.txt-621-			      <implicit address-dependency barrier>
 --
 Documentation/memory-barriers.txt=644=dependencies in a great many highly creative ways.
 --
 Documentation/memory-barriers.txt-651-	WRITE_ONCE(P, &B);
 Documentation/memory-barriers.txt:652:			      Q = READ_ONCE_OLD(P);
 Documentation/memory-barriers.txt-653-			      WRITE_ONCE(*Q, 5);
 --
 Documentation/memory-barriers.txt=656=Q with the store into *Q.  In other words, this outcome is prohibited,
 Documentation/memory-barriers.txt:657:even without an implicit address-dependency barrier of modern READ_ONCE():
 Documentation/memory-barriers.txt-658-
 --
 Documentation/memory-barriers.txt=689=Consider the following bit of code:
 Documentation/memory-barriers.txt-690-
 Documentation/memory-barriers.txt:691:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-692-	<implicit address-dependency barrier>
 --
 Documentation/memory-barriers.txt-694-		/* BUG: No address dependency!!! */
 Documentation/memory-barriers.txt:695:		p = READ_ONCE(b);
 Documentation/memory-barriers.txt-696-	}
 --
 Documentation/memory-barriers.txt=702=what's actually required is:
 Documentation/memory-barriers.txt-703-
 Documentation/memory-barriers.txt:704:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-705-	if (q) {
 Documentation/memory-barriers.txt-706-		<read barrier>
 Documentation/memory-barriers.txt:707:		p = READ_ONCE(b);
 Documentation/memory-barriers.txt-708-	}
 --
 Documentation/memory-barriers.txt=711=for load-store control dependencies, as in the following example:
 Documentation/memory-barriers.txt-712-
 Documentation/memory-barriers.txt:713:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-714-	if (q) {
 --
 Documentation/memory-barriers.txt=718=Control dependencies pair normally with other types of barriers.
 Documentation/memory-barriers.txt:719:That said, please note that neither READ_ONCE() nor WRITE_ONCE()
 Documentation/memory-barriers.txt:720:are optional! Without the READ_ONCE(), the compiler might combine the
 Documentation/memory-barriers.txt-721-load from 'a' with other loads from 'a'.  Without the WRITE_ONCE(),
 --
 Documentation/memory-barriers.txt=728=as follows:
 --
 Documentation/memory-barriers.txt-732-
 Documentation/memory-barriers.txt:733:So don't leave out the READ_ONCE().
 Documentation/memory-barriers.txt-734-
 --
 Documentation/memory-barriers.txt=736=branches of the "if" statement as follows:
 Documentation/memory-barriers.txt-737-
 Documentation/memory-barriers.txt:738:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-739-	if (q) {
 --
 Documentation/memory-barriers.txt=750=optimization levels:
 Documentation/memory-barriers.txt-751-
 Documentation/memory-barriers.txt:752:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-753-	barrier();
 --
 Documentation/memory-barriers.txt=768=memory barriers, for example, smp_store_release():
 Documentation/memory-barriers.txt-769-
 Documentation/memory-barriers.txt:770:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-771-	if (q) {
 --
 Documentation/memory-barriers.txt=780=ordering is guaranteed only when the stores differ, for example:
 Documentation/memory-barriers.txt-781-
 Documentation/memory-barriers.txt:782:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-783-	if (q) {
 --
 Documentation/memory-barriers.txt-790-
 Documentation/memory-barriers.txt:791:The initial READ_ONCE() is still required to prevent the compiler from
 Documentation/memory-barriers.txt-792-proving the value of 'a'.
 --
 Documentation/memory-barriers.txt=796=the needed conditional.  For example:
 Documentation/memory-barriers.txt-797-
 Documentation/memory-barriers.txt:798:	q = READ_ONCE(a);
 Documentation/memory-barriers.txt-799-	if (q % MAX) {
]
]

gemini-3-flash-preview

							input: 69956
							output: 27
							thoughts: 34
						

Okay, here's my take on this:

**Considering the Best Approach for Text Extraction**

Alright, so I need to extract some information from a file. My initial instinct, the old reliable, is to just reach for `grep` and pipe that through `read-file`. It's a quick and dirty solution, I know the syntax like the back of my hand, and it gets the job done. But… I'm thinking, is there a more elegant, efficient approach? I've been meaning to experiment more with `grepper`, and this might be the perfect opportunity. I could refine my search expression, and see if I can't wrangle a more precise result from that. It's about finding the right balance between expediency and a slightly more thoughtful, potentially cleaner solution. Let's see if I can get a better expression working first.

no matches

map[Expression:READ_ONCE.*lib/maple_tree.c]

map[Output:]

gemini-3-flash-preview

							input: 70212
							output: 19
							thoughts: 43
						

Okay, here's my take on those thoughts, framed as my own mental process:

**A Better Approach to That Search**

Hold on a second, I think I've spotted the issue. It looks like that initial `grepper` search is hitting all the files, and then the results are getting cut off. I need a more efficient way to handle this.

The usual approach is clearly not working here. I should be using `codesearch-dir-index` first to narrow down the scope. Then, once I have a targeted list of files, I can go in and use `read-file` to actually inspect their contents. This looks like the right path forward to get a complete picture.

map[Dir:lib]

map[Files:[alloc_tag.c argv_split.c ashldi3.c ashrdi3.c asn1_decoder.c asn1_encoder.c assoc_array.c atomic64.c atomic64_test.c audit.c base64.c bcd.c bch.c bitmap-str.c bitmap.c bitrev.c bootconfig-data.S bootconfig.c bsearch.c btree.c bucket_locks.c bug.c buildid.c bust_spinlocks.c cache_maint.c check_signature.c checksum.c closure.c clz_ctz.c clz_tab.c cmdline.c cmpdi2.c cmpxchg-emu.c codetag.c compat_audit.c cpu_rmap.c cpumask.c ctype.c debug_info.c debug_locks.c debugobjects.c dec_and_lock.c decompress.c decompress_bunzip2.c decompress_inflate.c decompress_unlz4.c decompress_unlzma.c decompress_unlzo.c decompress_unxz.c decompress_unzstd.c devmem_is_allowed.c devres.c dhry.h dhry_1.c dhry_2.c dhry_run.c digsig.c dump_stack.c dynamic_debug.c dynamic_queue_limits.c earlycpio.c errname.c error-inject.c errseq.c extable.c fault-inject-usercopy.c fault-inject.c fdt.c fdt_addresses.c fdt_empty_tree.c fdt_ro.c fdt_rw.c fdt_strerror.c fdt_sw.c fdt_wip.c find_bit.c find_bit_benchmark.c find_bit_benchmark_rust.rs flex_proportions.c fw_table.c genalloc.c generic-radix-tree.c glob.c group_cpus.c hexdump.c hweight.c idr.c inflate.c interval_tree.c interval_tree_test.c iomap.c iomap_copy.c iomem_copy.c iommu-helper.c iov_iter.c irq_poll.c irq_regs.c is_single_threaded.c kasprintf.c kfifo.c klist.c kobject.c kobject_uevent.c kstrtox.c kstrtox.h linear_ranges.c list_debug.c list_sort.c llist.c locking-selftest-hardirq.h locking-selftest-mutex.h locking-selftest-rlock-hardirq.h locking-selftest-rlock-softirq.h locking-selftest-rlock.h locking-selftest-rsem.h locking-selftest-rtmutex.h locking-selftest-softirq.h locking-selftest-spin-hardirq.h locking-selftest-spin-softirq.h locking-selftest-spin.h locking-selftest-wlock-hardirq.h locking-selftest-wlock-softirq.h locking-selftest-wlock.h locking-selftest-wsem.h locking-selftest.c lockref.c logic_iomem.c logic_pio.c lru_cache.c lshrdi3.c lwq.c maple_tree.c memcat_p.c memory-notifier-error-inject.c memregion.c memweight.c min_heap.c muldi3.c net_utils.c netdev-notifier-error-inject.c nlattr.c nmi_backtrace.c notifier-error-inject.c notifier-error-inject.h objagg.c objpool.c of-reconfig-notifier-error-inject.c oid_registry.c oid_registry_data.c once.c packing.c packing_test.c parman.c parser.c percpu-refcount.c percpu_counter.c percpu_test.c plist.c pm-notifier-error-inject.c polynomial.c radix-tree.c radix-tree.h random32.c ratelimit.c rbtree.c rbtree_test.c rcuref.c ref_tracker.c refcount.c rhashtable.c sbitmap.c scatterlist.c seq_buf.c sg_pool.c sg_split.c siphash.c smp_processor_id.c sort.c stackdepot.c stmp_device.c string.c string_helpers.c strncpy_from_user.c strnlen_user.c sys_info.c syscall.c test-kstrtox.c test_bitmap.c test_bitops.c test_bpf.c test_context-analysis.c test_debug_virtual.c test_dynamic_debug.c test_firmware.c test_fpu.h test_fpu_glue.c test_fpu_impl.c test_free_pages.c test_hexdump.c test_hmm.c test_hmm_uapi.h test_ida.c test_kho.c test_kmod.c test_lockup.c test_maple_tree.c test_memcat_p.c test_meminit.c test_module.c test_objagg.c test_objpool.c test_parman.c test_ref_tracker.c test_rhashtable.c test_static_key_base.c test_static_keys.c test_sysctl.c test_ubsan.c test_vmalloc.c test_xarray.c textsearch.c timerqueue.c trace_readwrite.c ts_bm.c ts_fsm.c ts_kmp.c ubsan.c ubsan.h ucmpdi2.c ucs2_string.c union_find.c usercopy.c uuid.c vsprintf.c win_minmax.c xarray.c xxhash.c] Subdirs:[842 crc crypto dim fonts kunit lz4 lzo math pldmfw raid6 reed_solomon test_fortify tests vdso xz zlib_deflate zlib_dfltcc zlib_inflate zstd]]

gemini-3-flash-preview

							input: 72630
							output: 23
							thoughts: 25
						

Here's my thought process, summarized as requested:

**Analyzing the `maple_tree.c` Codebase**

Alright, let's dive in. The task is clear: I need to get a handle on how this `maple_tree.c` file is structured, and specifically, to understand how it uses `READ_ONCE`. Given the likely performance considerations and the underlying data structures involved, I'm expecting this will be a crucial piece in understanding concurrent access patterns. I'll read through the file methodically, but I know where to look. I'll be searching for the macro `READ_ONCE` immediately. I'll be on the lookout for its placement. I should be able to get a solid overview of how the code's designers are managing memory ordering and ensuring data consistency. Then, I will focus on understanding what the code is reading and why `READ_ONCE` is required to access that data. This should reveal how the architecture is handling multi-threaded read operations, and it might also give me hints about the write operations and associated locking mechanisms used in the maple tree implementation. This will be an interesting exercise, given the inherent complexity of high-performance tree data structures.

map[File:lib/maple_tree.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0+
/*
 * Maple Tree implementation
 * Copyright (c) 2018-2022 Oracle Corporation
 * Authors: Liam R. Howlett <Liam.Howlett@oracle.com>
 *	    Matthew Wilcox <willy@infradead.org>
 * Copyright (c) 2023 ByteDance
 * Author: Peng Zhang <zhangpeng.00@bytedance.com>
 */

/*
 * DOC: Interesting implementation details of the Maple Tree
 *
 * Each node type has a number of slots for entries and a number of slots for
 * pivots.  In the case of dense nodes, the pivots are implied by the position
 * and are simply the slot index + the minimum of the node.
 *
 * In regular B-Tree terms, pivots are called keys.  The term pivot is used to
 * indicate that the tree is specifying ranges.  Pivots may appear in the
 * subtree with an entry attached to the value whereas keys are unique to a
 * specific position of a B-tree.  Pivot values are inclusive of the slot with
 * the same index.
 *
 *
 * The following illustrates the layout of a range64 nodes slots and pivots.
 *
 *
 *  Slots -> | 0 | 1 | 2 | ... | 12 | 13 | 14 | 15 |
 *           ┬   ┬   ┬   ┬     ┬    ┬    ┬    ┬    ┬
 *           │   │   │   │     │    │    │    │    └─ Implied maximum
 *           │   │   │   │     │    │    │    └─ Pivot 14
 *           │   │   │   │     │    │    └─ Pivot 13
 *           │   │   │   │     │    └─ Pivot 12
 *           │   │   │   │     └─ Pivot 11
 *           │   │   │   └─ Pivot 2
 *           │   │   └─ Pivot 1
 *           │   └─ Pivot 0
 *           └─  Implied minimum
 *
 * Slot contents:
 *  Internal (non-leaf) nodes contain pointers to other nodes.
 *  Leaf nodes contain entries.
 *
 * The location of interest is often referred to as an offset.  All offsets have
 * a slot, but the last offset has an implied pivot from the node above (or
 * UINT_MAX for the root node.
 *
 * Ranges complicate certain write activities.  When modifying any of
 * the B-tree variants, it is known that one entry will either be added or
 * deleted.  When modifying the Maple Tree, one store operation may overwrite
 * the entire data set, or one half of the tree, or the middle half of the tree.
 *
 */


#include <linux/maple_tree.h>
#include <linux/xarray.h>
#include <linux/types.h>
#include <linux/export.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <asm/barrier.h>

#define CREATE_TRACE_POINTS
#include <trace/events/maple_tree.h>

#define TP_FCT tracepoint_string(__func__)

/*
 * Kernel pointer hashing renders much of the maple tree dump useless as tagged
 * pointers get hashed to arbitrary values.
 *
 * If CONFIG_DEBUG_VM_MAPLE_TREE is set we are in a debug mode where it is
 * permissible to bypass this. Otherwise remain cautious and retain the hashing.
 *
 * Userland doesn't know about %px so also use %p there.
 */
#if defined(__KERNEL__) && defined(CONFIG_DEBUG_VM_MAPLE_TREE)
#define PTR_FMT "%px"
#else
#define PTR_FMT "%p"
#endif

#define MA_ROOT_PARENT 1

/*
 * Maple state flags
 * * MA_STATE_PREALLOC		- Preallocated nodes, WARN_ON allocation
 */
#define MA_STATE_PREALLOC	1

#define ma_parent_ptr(x) ((struct maple_pnode *)(x))
#define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT)
#define ma_mnode_ptr(x) ((struct maple_node *)(x))
#define ma_enode_ptr(x) ((struct maple_enode *)(x))
static struct kmem_cache *maple_node_cache;

#ifdef CONFIG_DEBUG_MAPLE_TREE
static const unsigned long mt_max[] = {
	[maple_dense]		= MAPLE_NODE_SLOTS,
	[maple_leaf_64]		= ULONG_MAX,
	[maple_range_64]	= ULONG_MAX,
	[maple_arange_64]	= ULONG_MAX,
};
#define mt_node_max(x) mt_max[mte_node_type(x)]
#endif

static const unsigned char mt_slots[] = {
	[maple_dense]		= MAPLE_NODE_SLOTS,
	[maple_leaf_64]		= MAPLE_RANGE64_SLOTS,
	[maple_range_64]	= MAPLE_RANGE64_SLOTS,
	[maple_arange_64]	= MAPLE_ARANGE64_SLOTS,
};
#define mt_slot_count(x) mt_slots[mte_node_type(x)]

static const unsigned char mt_pivots[] = {
	[maple_dense]		= 0,
	[maple_leaf_64]		= MAPLE_RANGE64_SLOTS - 1,
	[maple_range_64]	= MAPLE_RANGE64_SLOTS - 1,
	[maple_arange_64]	= MAPLE_ARANGE64_SLOTS - 1,
};
#define mt_pivot_count(x) mt_pivots[mte_node_type(x)]

static const unsigned char mt_min_slots[] = {
	[maple_dense]		= MAPLE_NODE_SLOTS / 2,
	[maple_leaf_64]		= (MAPLE_RANGE64_SLOTS / 2) - 2,
	[maple_range_64]	= (MAPLE_RANGE64_SLOTS / 2) - 2,
	[maple_arange_64]	= (MAPLE_ARANGE64_SLOTS / 2) - 1,
};
#define mt_min_slot_count(x) mt_min_slots[mte_node_type(x)]

#define MAPLE_BIG_NODE_SLOTS	(MAPLE_RANGE64_SLOTS * 2 + 2)
#define MAPLE_BIG_NODE_GAPS	(MAPLE_ARANGE64_SLOTS * 2 + 1)

struct maple_big_node {
	unsigned long pivot[MAPLE_BIG_NODE_SLOTS - 1];
	union {
		struct maple_enode *slot[MAPLE_BIG_NODE_SLOTS];
		struct {
			unsigned long padding[MAPLE_BIG_NODE_GAPS];
			unsigned long gap[MAPLE_BIG_NODE_GAPS];
		};
	};
	unsigned char b_end;
	enum maple_type type;
};

/*
 * The maple_subtree_state is used to build a tree to replace a segment of an
 * existing tree in a more atomic way.  Any walkers of the older tree will hit a
 * dead node and restart on updates.
 */
struct maple_subtree_state {
	struct ma_state *orig_l;	/* Original left side of subtree */
	struct ma_state *orig_r;	/* Original right side of subtree */
	struct ma_state *l;		/* New left side of subtree */
	struct ma_state *m;		/* New middle of subtree (rare) */
	struct ma_state *r;		/* New right side of subtree */
	struct ma_topiary *free;	/* nodes to be freed */
	struct ma_topiary *destroy;	/* Nodes to be destroyed (walked and freed) */
	struct maple_big_node *bn;
};

#ifdef CONFIG_KASAN_STACK
/* Prevent mas_wr_bnode() from exceeding the stack frame limit */
#define noinline_for_kasan noinline_for_stack
#else
#define noinline_for_kasan inline
#endif

/* Functions */
static inline struct maple_node *mt_alloc_one(gfp_t gfp)
{
	return kmem_cache_alloc(maple_node_cache, gfp);
}

static inline void mt_free_bulk(size_t size, void __rcu **nodes)
{
	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
}

static void mt_return_sheaf(struct slab_sheaf *sheaf)
{
	kmem_cache_return_sheaf(maple_node_cache, GFP_NOWAIT, sheaf);
}

static struct slab_sheaf *mt_get_sheaf(gfp_t gfp, int count)
{
	return kmem_cache_prefill_sheaf(maple_node_cache, gfp, count);
}

static int mt_refill_sheaf(gfp_t gfp, struct slab_sheaf **sheaf,
		unsigned int size)
{
	return kmem_cache_refill_sheaf(maple_node_cache, gfp, sheaf, size);
}

/*
 * ma_free_rcu() - Use rcu callback to free a maple node
 * @node: The node to free
 *
 * The maple tree uses the parent pointer to indicate this node is no longer in
 * use and will be freed.
 */
static void ma_free_rcu(struct maple_node *node)
{
	WARN_ON(node->parent != ma_parent_ptr(node));
	kfree_rcu(node, rcu);
}

static void mt_set_height(struct maple_tree *mt, unsigned char height)
{
	unsigned int new_flags = mt->ma_flags;

	new_flags &= ~MT_FLAGS_HEIGHT_MASK;
	MT_BUG_ON(mt, height > MAPLE_HEIGHT_MAX);
	new_flags |= height << MT_FLAGS_HEIGHT_OFFSET;
	mt->ma_flags = new_flags;
}

static unsigned int mas_mt_height(struct ma_state *mas)
{
	return mt_height(mas->tree);
}

static inline unsigned int mt_attr(struct maple_tree *mt)
{
	return mt->ma_flags & ~MT_FLAGS_HEIGHT_MASK;
}

static __always_inline enum maple_type mte_node_type(
		const struct maple_enode *entry)
{
	return ((unsigned long)entry >> MAPLE_NODE_TYPE_SHIFT) &
		MAPLE_NODE_TYPE_MASK;
}

static __always_inline bool ma_is_dense(const enum maple_type type)
{
	return type < maple_leaf_64;
}

static __always_inline bool ma_is_leaf(const enum maple_type type)
{
	return type < maple_range_64;
}

static __always_inline bool mte_is_leaf(const struct maple_enode *entry)
{
	return ma_is_leaf(mte_node_type(entry));
}

/*
 * We also reserve values with the bottom two bits set to '10' which are
 * below 4096
 */
static __always_inline bool mt_is_reserved(const void *entry)
{
	return ((unsigned long)entry < MAPLE_RESERVED_RANGE) &&
		xa_is_internal(entry);
}

static __always_inline void mas_set_err(struct ma_state *mas, long err)
{
	mas->node = MA_ERROR(err);
	mas->status = ma_error;
}

static __always_inline bool mas_is_ptr(const struct ma_state *mas)
{
	return mas->status == ma_root;
}

static __always_inline bool mas_is_start(const struct ma_state *mas)
{
	return mas->status == ma_start;
}

static __always_inline bool mas_is_none(const struct ma_state *mas)
{
	return mas->status == ma_none;
}

static __always_inline bool mas_is_paused(const struct ma_state *mas)
{
	return mas->status == ma_pause;
}

static __always_inline bool mas_is_overflow(struct ma_state *mas)
{
	return mas->status == ma_overflow;
}

static inline bool mas_is_underflow(struct ma_state *mas)
{
	return mas->status == ma_underflow;
}

static __always_inline struct maple_node *mte_to_node(
		const struct maple_enode *entry)
{
	return (struct maple_node *)((unsigned long)entry & ~MAPLE_NODE_MASK);
}

/*
 * mte_to_mat() - Convert a maple encoded node to a maple topiary node.
 * @entry: The maple encoded node
 *
 * Return: a maple topiary pointer
 */
static inline struct maple_topiary *mte_to_mat(const struct maple_enode *entry)
{
	return (struct maple_topiary *)
		((unsigned long)entry & ~MAPLE_NODE_MASK);
}

/*
 * mas_mn() - Get the maple state node.
 * @mas: The maple state
 *
 * Return: the maple node (not encoded - bare pointer).
 */
static inline struct maple_node *mas_mn(const struct ma_state *mas)
{
	return mte_to_node(mas->node);
}

/*
 * mte_set_node_dead() - Set a maple encoded node as dead.
 * @mn: The maple encoded node.
 */
static inline void mte_set_node_dead(struct maple_enode *mn)
{
	mte_to_node(mn)->parent = ma_parent_ptr(mte_to_node(mn));
	smp_wmb(); /* Needed for RCU */
}

/* Bit 1 indicates the root is a node */
#define MAPLE_ROOT_NODE			0x02
/* maple_type stored bit 3-6 */
#define MAPLE_ENODE_TYPE_SHIFT		0x03
/* Bit 2 means a NULL somewhere below */
#define MAPLE_ENODE_NULL		0x04

static inline struct maple_enode *mt_mk_node(const struct maple_node *node,
					     enum maple_type type)
{
	return (void *)((unsigned long)node |
			(type << MAPLE_ENODE_TYPE_SHIFT) | MAPLE_ENODE_NULL);
}

static inline void *mte_mk_root(const struct maple_enode *node)
{
	return (void *)((unsigned long)node | MAPLE_ROOT_NODE);
}

static inline void *mte_safe_root(const struct maple_enode *node)
{
	return (void *)((unsigned long)node & ~MAPLE_ROOT_NODE);
}

static inline void __maybe_unused *mte_set_full(const struct maple_enode *node)
{
	return (void *)((unsigned long)node & ~MAPLE_ENODE_NULL);
}

static inline void __maybe_unused *mte_clear_full(const struct maple_enode *node)
{
	return (void *)((unsigned long)node | MAPLE_ENODE_NULL);
}

static inline bool __maybe_unused mte_has_null(const struct maple_enode *node)
{
	return (unsigned long)node & MAPLE_ENODE_NULL;
}

static __always_inline bool ma_is_root(struct maple_node *node)
{
	return ((unsigned long)node->parent & MA_ROOT_PARENT);
}

static __always_inline bool mte_is_root(const struct maple_enode *node)
{
	return ma_is_root(mte_to_node(node));
}

static inline bool mas_is_root_limits(const struct ma_state *mas)
{
	return !mas->min && mas->max == ULONG_MAX;
}

static __always_inline bool mt_is_alloc(struct maple_tree *mt)
{
	return (mt->ma_flags & MT_FLAGS_ALLOC_RANGE);
}

/*
 * The Parent Pointer
 * Excluding root, the parent pointer is 256B aligned like all other tree nodes.
 * When storing a 32 or 64 bit values, the offset can fit into 5 bits.  The 16
 * bit values need an extra bit to store the offset.  This extra bit comes from
 * a reuse of the last bit in the node type.  This is possible by using bit 1 to
 * indicate if bit 2 is part of the type or the slot.
 *
 * Node types:
 *  0b??1 = Root
 *  0b?00 = 16 bit nodes
 *  0b010 = 32 bit nodes
 *  0b110 = 64 bit nodes
 *
 * Slot size and alignment
 *  0b??1 : Root
 *  0b?00 : 16 bit values, type in 0-1, slot in 2-7
 *  0b010 : 32 bit values, type in 0-2, slot in 3-7
 *  0b110 : 64 bit values, type in 0-2, slot in 3-7
 */

#define MAPLE_PARENT_ROOT		0x01

#define MAPLE_PARENT_SLOT_SHIFT		0x03
#define MAPLE_PARENT_SLOT_MASK		0xF8

#define MAPLE_PARENT_16B_SLOT_SHIFT	0x02
#define MAPLE_PARENT_16B_SLOT_MASK	0xFC

#define MAPLE_PARENT_RANGE64		0x06
#define MAPLE_PARENT_RANGE32		0x02
#define MAPLE_PARENT_NOT_RANGE16	0x02

/*
 * mte_parent_shift() - Get the parent shift for the slot storage.
 * @parent: The parent pointer cast as an unsigned long
 * Return: The shift into that pointer to the star to of the slot
 */
static inline unsigned long mte_parent_shift(unsigned long parent)
{
	/* Note bit 1 == 0 means 16B */
	if (likely(parent & MAPLE_PARENT_NOT_RANGE16))
		return MAPLE_PARENT_SLOT_SHIFT;

	return MAPLE_PARENT_16B_SLOT_SHIFT;
}

/*
 * mte_parent_slot_mask() - Get the slot mask for the parent.
 * @parent: The parent pointer cast as an unsigned long.
 * Return: The slot mask for that parent.
 */
static inline unsigned long mte_parent_slot_mask(unsigned long parent)
{
	/* Note bit 1 == 0 means 16B */
	if (likely(parent & MAPLE_PARENT_NOT_RANGE16))
		return MAPLE_PARENT_SLOT_MASK;

	return MAPLE_PARENT_16B_SLOT_MASK;
}

/*
 * mas_parent_type() - Return the maple_type of the parent from the stored
 * parent type.
 * @mas: The maple state
 * @enode: The maple_enode to extract the parent's enum
 * Return: The node->parent maple_type
 */
static inline
enum maple_type mas_parent_type(struct ma_state *mas, struct maple_enode *enode)
{
	unsigned long p_type;

	p_type = (unsigned long)mte_to_node(enode)->parent;
	if (WARN_ON(p_type & MAPLE_PARENT_ROOT))
		return 0;

	p_type &= MAPLE_NODE_MASK;
	p_type &= ~mte_parent_slot_mask(p_type);
	switch (p_type) {
	case MAPLE_PARENT_RANGE64: /* or MAPLE_PARENT_ARANGE64 */
		if (mt_is_alloc(mas->tree))
			return maple_arange_64;
		return maple_range_64;
	}

	return 0;
}

/*
 * mas_set_parent() - Set the parent node and encode the slot
 * @mas: The maple state
 * @enode: The encoded maple node.
 * @parent: The encoded maple node that is the parent of @enode.
 * @slot: The slot that @enode resides in @parent.
 *
 * Slot number is encoded in the enode->parent bit 3-6 or 2-6, depending on the
 * parent type.
 */
static inline
void mas_set_parent(struct ma_state *mas, struct maple_enode *enode,
		    const struct maple_enode *parent, unsigned char slot)
{
	unsigned long val = (unsigned long)parent;
	unsigned long shift;
	unsigned long type;
	enum maple_type p_type = mte_node_type(parent);

	MAS_BUG_ON(mas, p_type == maple_dense);
	MAS_BUG_ON(mas, p_type == maple_leaf_64);

	switch (p_type) {
	case maple_range_64:
	case maple_arange_64:
		shift = MAPLE_PARENT_SLOT_SHIFT;
		type = MAPLE_PARENT_RANGE64;
		break;
	default:
	case maple_dense:
	case maple_leaf_64:
		shift = type = 0;
		break;
	}

	val &= ~MAPLE_NODE_MASK; /* Clear all node metadata in parent */
	val |= (slot << shift) | type;
	mte_to_node(enode)->parent = ma_parent_ptr(val);
}

/*
 * mte_parent_slot() - get the parent slot of @enode.
 * @enode: The encoded maple node.
 *
 * Return: The slot in the parent node where @enode resides.
 */
static __always_inline
unsigned int mte_parent_slot(const struct maple_enode *enode)
{
	unsigned long val = (unsigned long)mte_to_node(enode)->parent;

	if (unlikely(val & MA_ROOT_PARENT))
		return 0;

	/*
	 * Okay to use MAPLE_PARENT_16B_SLOT_MASK as the last bit will be lost
	 * by shift if the parent shift is MAPLE_PARENT_SLOT_SHIFT
	 */
	return (val & MAPLE_PARENT_16B_SLOT_MASK) >> mte_parent_shift(val);
}

/*
 * mte_parent() - Get the parent of @node.
 * @enode: The encoded maple node.
 *
 * Return: The parent maple node.
 */
static __always_inline
struct maple_node *mte_parent(const struct maple_enode *enode)
{
	return (void *)((unsigned long)
			(mte_to_node(enode)->parent) & ~MAPLE_NODE_MASK);
}

/*
 * ma_dead_node() - check if the @enode is dead.
 * @enode: The encoded maple node
 *
 * Return: true if dead, false otherwise.
 */
static __always_inline bool ma_dead_node(const struct maple_node *node)
{
	struct maple_node *parent;

	/* Do not reorder reads from the node prior to the parent check */
	smp_rmb();
	parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK);
	return (parent == node);
}

/*
 * mte_dead_node() - check if the @enode is dead.
 * @enode: The encoded maple node
 *
 * Return: true if dead, false otherwise.
 */
static __always_inline bool mte_dead_node(const struct maple_enode *enode)
{
	struct maple_node *node;

	node = mte_to_node(enode);
	return ma_dead_node(node);
}

/*
 * ma_pivots() - Get a pointer to the maple node pivots.
 * @node: the maple node
 * @type: the node type
 *
 * In the event of a dead node, this array may be %NULL
 *
 * Return: A pointer to the maple node pivots
 */
static inline unsigned long *ma_pivots(struct maple_node *node,
					   enum maple_type type)
{
	switch (type) {
	case maple_arange_64:
		return node->ma64.pivot;
	case maple_range_64:
	case maple_leaf_64:
		return node->mr64.pivot;
	case maple_dense:
		return NULL;
	}
	return NULL;
}

/*
 * ma_gaps() - Get a pointer to the maple node gaps.
 * @node: the maple node
 * @type: the node type
 *
 * Return: A pointer to the maple node gaps
 */
static inline unsigned long *ma_gaps(struct maple_node *node,
				     enum maple_type type)
{
	switch (type) {
	case maple_arange_64:
		return node->ma64.gap;
	case maple_range_64:
	case maple_leaf_64:
	case maple_dense:
		return NULL;
	}
	return NULL;
}

/*
 * mas_safe_pivot() - get the pivot at @piv or mas->max.
 * @mas: The maple state
 * @pivots: The pointer to the maple node pivots
 * @piv: The pivot to fetch
 * @type: The maple node type
 *
 * Return: The pivot at @piv within the limit of the @pivots array, @mas->max
 * otherwise.
 */
static __always_inline unsigned long
mas_safe_pivot(const struct ma_state *mas, unsigned long *pivots,
	       unsigned char piv, enum maple_type type)
{
	if (piv >= mt_pivots[type])
		return mas->max;

	return pivots[piv];
}

/*
 * mas_safe_min() - Return the minimum for a given offset.
 * @mas: The maple state
 * @pivots: The pointer to the maple node pivots
 * @offset: The offset into the pivot array
 *
 * Return: The minimum range value that is contained in @offset.
 */
static inline unsigned long
mas_safe_min(struct ma_state *mas, unsigned long *pivots, unsigned char offset)
{
	if (likely(offset))
		return pivots[offset - 1] + 1;

	return mas->min;
}

/*
 * mte_set_pivot() - Set a pivot to a value in an encoded maple node.
 * @mn: The encoded maple node
 * @piv: The pivot offset
 * @val: The value of the pivot
 */
static inline void mte_set_pivot(struct maple_enode *mn, unsigned char piv,
				unsigned long val)
{
	struct maple_node *node = mte_to_node(mn);
	enum maple_type type = mte_node_type(mn);

	BUG_ON(piv >= mt_pivots[type]);
	switch (type) {
	case maple_range_64:
	case maple_leaf_64:
		node->mr64.pivot[piv] = val;
		break;
	case maple_arange_64:
		node->ma64.pivot[piv] = val;
		break;
	case maple_dense:
		break;
	}

}

/*
 * ma_slots() - Get a pointer to the maple node slots.
 * @mn: The maple node
 * @mt: The maple node type
 *
 * Return: A pointer to the maple node slots
 */
static inline void __rcu **ma_slots(struct maple_node *mn, enum maple_type mt)
{
	switch (mt) {
	case maple_arange_64:
		return mn->ma64.slot;
	case maple_range_64:
	case maple_leaf_64:
		return mn->mr64.slot;
	case maple_dense:
		return mn->slot;
	}

	return NULL;
}

static inline bool mt_write_locked(const struct maple_tree *mt)
{
	return mt_external_lock(mt) ? mt_write_lock_is_held(mt) :
		lockdep_is_held(&mt->ma_lock);
}

static __always_inline bool mt_locked(const struct maple_tree *mt)
{
	return mt_external_lock(mt) ? mt_lock_is_held(mt) :
		lockdep_is_held(&mt->ma_lock);
}

static __always_inline void *mt_slot(const struct maple_tree *mt,
		void __rcu **slots, unsigned char offset)
{
	return rcu_dereference_check(slots[offset], mt_locked(mt));
}

static __always_inline void *mt_slot_locked(struct maple_tree *mt,
		void __rcu **slots, unsigned char offset)
{
	return rcu_dereference_protected(slots[offset], mt_write_locked(mt));
}
/*
 * mas_slot_locked() - Get the slot value when holding the maple tree lock.
 * @mas: The maple state
 * @slots: The pointer to the slots
 * @offset: The offset into the slots array to fetch
 *
 * Return: The entry stored in @slots at the @offset.
 */
static __always_inline void *mas_slot_locked(struct ma_state *mas,
		void __rcu **slots, unsigned char offset)
{
	return mt_slot_locked(mas->tree, slots, offset);
}

/*
 * mas_slot() - Get the slot value when not holding the maple tree lock.
 * @mas: The maple state
 * @slots: The pointer to the slots
 * @offset: The offset into the slots array to fetch
 *
 * Return: The entry stored in @slots at the @offset
 */
static __always_inline void *mas_slot(struct ma_state *mas, void __rcu **slots,
		unsigned char offset)
{
	return mt_slot(mas->tree, slots, offset);
}

/*
 * mas_root() - Get the maple tree root.
 * @mas: The maple state.
 *
 * Return: The pointer to the root of the tree
 */
static __always_inline void *mas_root(struct ma_state *mas)
{
	return rcu_dereference_check(mas->tree->ma_root, mt_locked(mas->tree));
}

static inline void *mt_root_locked(struct maple_tree *mt)
{
	return rcu_dereference_protected(mt->ma_root, mt_write_locked(mt));
}

/*
 * mas_root_locked() - Get the maple tree root when holding the maple tree lock.
 * @mas: The maple state.
 *
 * Return: The pointer to the root of the tree
 */
static inline void *mas_root_locked(struct ma_state *mas)
{
	return mt_root_locked(mas->tree);
}

static inline struct maple_metadata *ma_meta(struct maple_node *mn,
					     enum maple_type mt)
{
	switch (mt) {
	case maple_arange_64:
		return &mn->ma64.meta;
	default:
		return &mn->mr64.meta;
	}
}

/*
 * ma_set_meta() - Set the metadata information of a node.
 * @mn: The maple node
 * @mt: The maple node type
 * @offset: The offset of the highest sub-gap in this node.
 * @end: The end of the data in this node.
 */
static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt,
			       unsigned char offset, unsigned char end)
{
	struct maple_metadata *meta = ma_meta(mn, mt);

	meta->gap = offset;
	meta->end = end;
}

/*
 * mt_clear_meta() - clear the metadata information of a node, if it exists
 * @mt: The maple tree
 * @mn: The maple node
 * @type: The maple node type
 */
static inline void mt_clear_meta(struct maple_tree *mt, struct maple_node *mn,
				  enum maple_type type)
{
	struct maple_metadata *meta;
	unsigned long *pivots;
	void __rcu **slots;
	void *next;

	switch (type) {
	case maple_range_64:
		pivots = mn->mr64.pivot;
		if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) {
			slots = mn->mr64.slot;
			next = mt_slot_locked(mt, slots,
					      MAPLE_RANGE64_SLOTS - 1);
			if (unlikely((mte_to_node(next) &&
				      mte_node_type(next))))
				return; /* no metadata, could be node */
		}
		fallthrough;
	case maple_arange_64:
		meta = ma_meta(mn, type);
		break;
	default:
		return;
	}

	meta->gap = 0;
	meta->end = 0;
}

/*
 * ma_meta_end() - Get the data end of a node from the metadata
 * @mn: The maple node
 * @mt: The maple node type
 */
static inline unsigned char ma_meta_end(struct maple_node *mn,
					enum maple_type mt)
{
	struct maple_metadata *meta = ma_meta(mn, mt);

	return meta->end;
}

/*
 * ma_meta_gap() - Get the largest gap location of a node from the metadata
 * @mn: The maple node
 */
static inline unsigned char ma_meta_gap(struct maple_node *mn)
{
	return mn->ma64.meta.gap;
}

/*
 * ma_set_meta_gap() - Set the largest gap location in a nodes metadata
 * @mn: The maple node
 * @mt: The maple node type
 * @offset: The location of the largest gap.
 */
static inline void ma_set_meta_gap(struct maple_node *mn, enum maple_type mt,
				   unsigned char offset)
{

	struct maple_metadata *meta = ma_meta(mn, mt);

	meta->gap = offset;
}

/*
 * mat_add() - Add a @dead_enode to the ma_topiary of a list of dead nodes.
 * @mat: the ma_topiary, a linked list of dead nodes.
 * @dead_enode: the node to be marked as dead and added to the tail of the list
 *
 * Add the @dead_enode to the linked list in @mat.
 */
static inline void mat_add(struct ma_topiary *mat,
			   struct maple_enode *dead_enode)
{
	mte_set_node_dead(dead_enode);
	mte_to_mat(dead_enode)->next = NULL;
	if (!mat->tail) {
		mat->tail = mat->head = dead_enode;
		return;
	}

	mte_to_mat(mat->tail)->next = dead_enode;
	mat->tail = dead_enode;
}

static void mt_free_walk(struct rcu_head *head);
static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
			    bool free);
/*
 * mas_mat_destroy() - Free all nodes and subtrees in a dead list.
 * @mas: the maple state
 * @mat: the ma_topiary linked list of dead nodes to free.
 *
 * Destroy walk a dead list.
 */
static void mas_mat_destroy(struct ma_state *mas, struct ma_topiary *mat)
{
	struct maple_enode *next;
	struct maple_node *node;
	bool in_rcu = mt_in_rcu(mas->tree);

	while (mat->head) {
		next = mte_to_mat(mat->head)->next;
		node = mte_to_node(mat->head);
		mt_destroy_walk(mat->head, mas->tree, !in_rcu);
		if (in_rcu)
			call_rcu(&node->rcu, mt_free_walk);
		mat->head = next;
	}
}
/*
 * mas_descend() - Descend into the slot stored in the ma_state.
 * @mas: the maple state.
 *
 * Note: Not RCU safe, only use in write side or debug code.
 */
static inline void mas_descend(struct ma_state *mas)
{
	enum maple_type type;
	unsigned long *pivots;
	struct maple_node *node;
	void __rcu **slots;

	node = mas_mn(mas);
	type = mte_node_type(mas->node);
	pivots = ma_pivots(node, type);
	slots = ma_slots(node, type);

	if (mas->offset)
		mas->min = pivots[mas->offset - 1] + 1;
	mas->max = mas_safe_pivot(mas, pivots, mas->offset, type);
	mas->node = mas_slot(mas, slots, mas->offset);
}

/*
 * mas_ascend() - Walk up a level of the tree.
 * @mas: The maple state
 *
 * Sets the @mas->max and @mas->min for the parent node of mas->node.  This
 * may cause several levels of walking up to find the correct min and max.
 * May find a dead node which will cause a premature return.
 * Return: 1 on dead node, 0 otherwise
 */
static int mas_ascend(struct ma_state *mas)
{
	struct maple_enode *p_enode; /* parent enode. */
	struct maple_enode *a_enode; /* ancestor enode. */
	struct maple_node *a_node; /* ancestor node. */
	struct maple_node *p_node; /* parent node. */
	unsigned char a_slot;
	enum maple_type a_type;
	unsigned long min, max;
	unsigned long *pivots;
	bool set_max = false, set_min = false;

	a_node = mas_mn(mas);
	if (ma_is_root(a_node)) {
		mas->offset = 0;
		return 0;
	}

	p_node = mte_parent(mas->node);
	if (unlikely(a_node == p_node))
		return 1;

	a_type = mas_parent_type(mas, mas->node);
	mas->offset = mte_parent_slot(mas->node);
	a_enode = mt_mk_node(p_node, a_type);

	/* Check to make sure all parent information is still accurate */
	if (p_node != mte_parent(mas->node))
		return 1;

	mas->node = a_enode;

	if (mte_is_root(a_enode)) {
		mas->max = ULONG_MAX;
		mas->min = 0;
		return 0;
	}

	min = 0;
	max = ULONG_MAX;

	/*
	 * !mas->offset implies that parent node min == mas->min.
	 * mas->offset > 0 implies that we need to walk up to find the
	 * implied pivot min.
	 */
	if (!mas->offset) {
		min = mas->min;
		set_min = true;
	}

	if (mas->max == ULONG_MAX)
		set_max = true;

	do {
		p_enode = a_enode;
		a_type = mas_parent_type(mas, p_enode);
		a_node = mte_parent(p_enode);
		a_slot = mte_parent_slot(p_enode);
		a_enode = mt_mk_node(a_node, a_type);
		pivots = ma_pivots(a_node, a_type);

		if (unlikely(ma_dead_node(a_node)))
			return 1;

		if (!set_min && a_slot) {
			set_min = true;
			min = pivots[a_slot - 1] + 1;
		}

		if (!set_max && a_slot < mt_pivots[a_type]) {
			set_max = true;
			max = pivots[a_slot];
		}

		if (unlikely(ma_dead_node(a_node)))
			return 1;

		if (unlikely(ma_is_root(a_node)))
			break;

	} while (!set_min || !set_max);

	mas->max = max;
	mas->min = min;
	return 0;
}

/*
 * mas_pop_node() - Get a previously allocated maple node from the maple state.
 * @mas: The maple state
 *
 * Return: A pointer to a maple node.
 */
static __always_inline struct maple_node *mas_pop_node(struct ma_state *mas)
{
	struct maple_node *ret;

	if (mas->alloc) {
		ret = mas->alloc;
		mas->alloc = NULL;
		goto out;
	}

	if (WARN_ON_ONCE(!mas->sheaf))
		return NULL;

	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);

out:
	memset(ret, 0, sizeof(*ret));
	return ret;
}

/*
 * mas_alloc_nodes() - Allocate nodes into a maple state
 * @mas: The maple state
 * @gfp: The GFP Flags
 */
static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
{
	if (!mas->node_request)
		return;

	if (mas->node_request == 1) {
		if (mas->sheaf)
			goto use_sheaf;

		if (mas->alloc)
			return;

		mas->alloc = mt_alloc_one(gfp);
		if (!mas->alloc)
			goto error;

		mas->node_request = 0;
		return;
	}

use_sheaf:
	if (unlikely(mas->alloc)) {
		kfree(mas->alloc);
		mas->alloc = NULL;
	}

	if (mas->sheaf) {
		unsigned long refill;

		refill = mas->node_request;
		if (kmem_cache_sheaf_size(mas->sheaf) >= refill) {
			mas->node_request = 0;
			return;
		}

		if (mt_refill_sheaf(gfp, &mas->sheaf, refill))
			goto error;

		mas->node_request = 0;
		return;
	}

	mas->sheaf = mt_get_sheaf(gfp, mas->node_request);
	if (likely(mas->sheaf)) {
		mas->node_request = 0;
		return;
	}

error:
	mas_set_err(mas, -ENOMEM);
}

static inline void mas_empty_nodes(struct ma_state *mas)
{
	mas->node_request = 0;
	if (mas->sheaf) {
		mt_return_sheaf(mas->sheaf);
		mas->sheaf = NULL;
	}

	if (mas->alloc) {
		kfree(mas->alloc);
		mas->alloc = NULL;
	}
}

/*
 * mas_free() - Free an encoded maple node
 * @mas: The maple state
 * @used: The encoded maple node to free.
 *
 * Uses rcu free if necessary, pushes @used back on the maple state allocations
 * otherwise.
 */
static inline void mas_free(struct ma_state *mas, struct maple_enode *used)
{
	ma_free_rcu(mte_to_node(used));
}

/*
 * mas_start() - Sets up maple state for operations.
 * @mas: The maple state.
 *
 * If mas->status == ma_start, then set the min, max and depth to
 * defaults.
 *
 * Return:
 * - If mas->node is an error or not mas_start, return NULL.
 * - If it's an empty tree:     NULL & mas->status == ma_none
 * - If it's a single entry:    The entry & mas->status == ma_root
 * - If it's a tree:            NULL & mas->status == ma_active
 */
static inline struct maple_enode *mas_start(struct ma_state *mas)
{
	if (likely(mas_is_start(mas))) {
		struct maple_enode *root;

		mas->min = 0;
		mas->max = ULONG_MAX;

retry:
		mas->depth = 0;
		root = mas_root(mas);
		/* Tree with nodes */
		if (likely(xa_is_node(root))) {
			mas->depth = 0;
			mas->status = ma_active;
			mas->node = mte_safe_root(root);
			mas->offset = 0;
			if (mte_dead_node(mas->node))
				goto retry;

			return NULL;
		}

		mas->node = NULL;
		/* empty tree */
		if (unlikely(!root)) {
			mas->status = ma_none;
			mas->offset = MAPLE_NODE_SLOTS;
			return NULL;
		}

		/* Single entry tree */
		mas->status = ma_root;
		mas->offset = MAPLE_NODE_SLOTS;

		/* Single entry tree. */
		if (mas->index > 0)
			return NULL;

		return root;
	}

	return NULL;
}

/*
 * ma_data_end() - Find the end of the data in a node.
 * @node: The maple node
 * @type: The maple node type
 * @pivots: The array of pivots in the node
 * @max: The maximum value in the node
 *
 * Uses metadata to find the end of the data when possible.
 * Return: The zero indexed last slot with data (may be null).
 */
static __always_inline unsigned char ma_data_end(struct maple_node *node,
		enum maple_type type, unsigned long *pivots, unsigned long max)
{
	unsigned char offset;

	if (!pivots)
		return 0;

	if (type == maple_arange_64)
		return ma_meta_end(node, type);

	offset = mt_pivots[type] - 1;
	if (likely(!pivots[offset]))
		return ma_meta_end(node, type);

	if (likely(pivots[offset] == max))
		return offset;

	return mt_pivots[type];
}

/*
 * mas_data_end() - Find the end of the data (slot).
 * @mas: the maple state
 *
 * This method is optimized to check the metadata of a node if the node type
 * supports data end metadata.
 *
 * Return: The zero indexed last slot with data (may be null).
 */
static inline unsigned char mas_data_end(struct ma_state *mas)
{
	enum maple_type type;
	struct maple_node *node;
	unsigned char offset;
	unsigned long *pivots;

	type = mte_node_type(mas->node);
	node = mas_mn(mas);
	if (type == maple_arange_64)
		return ma_meta_end(node, type);

	pivots = ma_pivots(node, type);
	if (unlikely(ma_dead_node(node)))
		return 0;

	offset = mt_pivots[type] - 1;
	if (likely(!pivots[offset]))
		return ma_meta_end(node, type);

	if (likely(pivots[offset] == mas->max))
		return offset;

	return mt_pivots[type];
}

/*
 * mas_leaf_max_gap() - Returns the largest gap in a leaf node
 * @mas: the maple state
 *
 * Return: The maximum gap in the leaf.
 */
static unsigned long mas_leaf_max_gap(struct ma_state *mas)
{
	enum maple_type mt;
	unsigned long pstart, gap, max_gap;
	struct maple_node *mn;
	unsigned long *pivots;
	void __rcu **slots;
	unsigned char i;
	unsigned char max_piv;

	mt = mte_node_type(mas->node);
	mn = mas_mn(mas);
	slots = ma_slots(mn, mt);
	max_gap = 0;
	if (unlikely(ma_is_dense(mt))) {
		gap = 0;
		for (i = 0; i < mt_slots[mt]; i++) {
			if (slots[i]) {
				if (gap > max_gap)
					max_gap = gap;
				gap = 0;
			} else {
				gap++;
			}
		}
		if (gap > max_gap)
			max_gap = gap;
		return max_gap;
	}

	/*
	 * Check the first implied pivot optimizes the loop below and slot 1 may
	 * be skipped if there is a gap in slot 0.
	 */
	pivots = ma_pivots(mn, mt);
	if (likely(!slots[0])) {
		max_gap = pivots[0] - mas->min + 1;
		i = 2;
	} else {
		i = 1;
	}

	/* reduce max_piv as the special case is checked before the loop */
	max_piv = ma_data_end(mn, mt, pivots, mas->max) - 1;
	/*
	 * Check end implied pivot which can only be a gap on the right most
	 * node.
	 */
	if (unlikely(mas->max == ULONG_MAX) && !slots[max_piv + 1]) {
		gap = ULONG_MAX - pivots[max_piv];
		if (gap > max_gap)
			max_gap = gap;

		if (max_gap > pivots[max_piv] - mas->min)
			return max_gap;
	}

	for (; i <= max_piv; i++) {
		/* data == no gap. */
		if (likely(slots[i]))
			continue;

		pstart = pivots[i - 1];
		gap = pivots[i] - pstart;
		if (gap > max_gap)
			max_gap = gap;

		/* There cannot be two gaps in a row. */
		i++;
	}
	return max_gap;
}

/*
 * ma_max_gap() - Get the maximum gap in a maple node (non-leaf)
 * @node: The maple node
 * @gaps: The pointer to the gaps
 * @mt: The maple node type
 * @off: Pointer to store the offset location of the gap.
 *
 * Uses the metadata data end to scan backwards across set gaps.
 *
 * Return: The maximum gap value
 */
static inline unsigned long
ma_max_gap(struct maple_node *node, unsigned long *gaps, enum maple_type mt,
	    unsigned char *off)
{
	unsigned char offset, i;
	unsigned long max_gap = 0;

	i = offset = ma_meta_end(node, mt);
	do {
		if (gaps[i] > max_gap) {
			max_gap = gaps[i];
			offset = i;
		}
	} while (i--);

	*off = offset;
	return max_gap;
}

/*
 * mas_max_gap() - find the largest gap in a non-leaf node and set the slot.
 * @mas: The maple state.
 *
 * Return: The gap value.
 */
static inline unsigned long mas_max_gap(struct ma_state *mas)
{
	unsigned long *gaps;
	unsigned char offset;
	enum maple_type mt;
	struct maple_node *node;

	mt = mte_node_type(mas->node);
	if (ma_is_leaf(mt))
		return mas_leaf_max_gap(mas);

	node = mas_mn(mas);
	MAS_BUG_ON(mas, mt != maple_arange_64);
	offset = ma_meta_gap(node);
	gaps = ma_gaps(node, mt);
	return gaps[offset];
}

/*
 * mas_parent_gap() - Set the parent gap and any gaps above, as needed
 * @mas: The maple state
 * @offset: The gap offset in the parent to set
 * @new: The new gap value.
 *
 * Set the parent gap then continue to set the gap upwards, using the metadata
 * of the parent to see if it is necessary to check the node above.
 */
static inline void mas_parent_gap(struct ma_state *mas, unsigned char offset,
		unsigned long new)
{
	unsigned long meta_gap = 0;
	struct maple_node *pnode;
	struct maple_enode *penode;
	unsigned long *pgaps;
	unsigned char meta_offset;
	enum maple_type pmt;

	pnode = mte_parent(mas->node);
	pmt = mas_parent_type(mas, mas->node);
	penode = mt_mk_node(pnode, pmt);
	pgaps = ma_gaps(pnode, pmt);

ascend:
	MAS_BUG_ON(mas, pmt != maple_arange_64);
	meta_offset = ma_meta_gap(pnode);
	meta_gap = pgaps[meta_offset];

	pgaps[offset] = new;

	if (meta_gap == new)
		return;

	if (offset != meta_offset) {
		if (meta_gap > new)
			return;

		ma_set_meta_gap(pnode, pmt, offset);
	} else if (new < meta_gap) {
		new = ma_max_gap(pnode, pgaps, pmt, &meta_offset);
		ma_set_meta_gap(pnode, pmt, meta_offset);
	}

	if (ma_is_root(pnode))
		return;

	/* Go to the parent node. */
	pnode = mte_parent(penode);
	pmt = mas_parent_type(mas, penode);
	pgaps = ma_gaps(pnode, pmt);
	offset = mte_parent_slot(penode);
	penode = mt_mk_node(pnode, pmt);
	goto ascend;
}

/*
 * mas_update_gap() - Update a nodes gaps and propagate up if necessary.
 * @mas: the maple state.
 */
static inline void mas_update_gap(struct ma_state *mas)
{
	unsigned char pslot;
	unsigned long p_gap;
	unsigned long max_gap;

	if (!mt_is_alloc(mas->tree))
		return;

	if (mte_is_root(mas->node))
		return;

	max_gap = mas_max_gap(mas);

	pslot = mte_parent_slot(mas->node);
	p_gap = ma_gaps(mte_parent(mas->node),
			mas_parent_type(mas, mas->node))[pslot];

	if (p_gap != max_gap)
		mas_parent_gap(mas, pslot, max_gap);
}

/*
 * mas_adopt_children() - Set the parent pointer of all nodes in @parent to
 * @parent with the slot encoded.
 * @mas: the maple state (for the tree)
 * @parent: the maple encoded node containing the children.
 */
static inline void mas_adopt_children(struct ma_state *mas,
		struct maple_enode *parent)
{
	enum maple_type type = mte_node_type(parent);
	struct maple_node *node = mte_to_node(parent);
	void __rcu **slots = ma_slots(node, type);
	unsigned long *pivots = ma_pivots(node, type);
	struct maple_enode *child;
	unsigned char offset;

	offset = ma_data_end(node, type, pivots, mas->max);
	do {
		child = mas_slot_locked(mas, slots, offset);
		mas_set_parent(mas, child, parent, offset);
	} while (offset--);
}

/*
 * mas_put_in_tree() - Put a new node in the tree, smp_wmb(), and mark the old
 * node as dead.
 * @mas: the maple state with the new node
 * @old_enode: The old maple encoded node to replace.
 * @new_height: if we are inserting a root node, update the height of the tree
 */
static inline void mas_put_in_tree(struct ma_state *mas,
		struct maple_enode *old_enode, char new_height)
	__must_hold(mas->tree->ma_lock)
{
	unsigned char offset;
	void __rcu **slots;

	if (mte_is_root(mas->node)) {
		mas_mn(mas)->parent = ma_parent_ptr(mas_tree_parent(mas));
		rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
		mt_set_height(mas->tree, new_height);
	} else {

		offset = mte_parent_slot(mas->node);
		slots = ma_slots(mte_parent(mas->node),
				 mas_parent_type(mas, mas->node));
		rcu_assign_pointer(slots[offset], mas->node);
	}

	mte_set_node_dead(old_enode);
}

/*
 * mas_replace_node() - Replace a node by putting it in the tree, marking it
 * dead, and freeing it.
 * the parent encoding to locate the maple node in the tree.
 * @mas: the ma_state with @mas->node pointing to the new node.
 * @old_enode: The old maple encoded node.
 * @new_height: The new height of the tree as a result of the operation
 */
static inline void mas_replace_node(struct ma_state *mas,
		struct maple_enode *old_enode, unsigned char new_height)
	__must_hold(mas->tree->ma_lock)
{
	mas_put_in_tree(mas, old_enode, new_height);
	mas_free(mas, old_enode);
}

/*
 * mas_find_child() - Find a child who has the parent @mas->node.
 * @mas: the maple state with the parent.
 * @child: the maple state to store the child.
 */
static inline bool mas_find_child(struct ma_state *mas, struct ma_state *child)
	__must_hold(mas->tree->ma_lock)
{
	enum maple_type mt;
	unsigned char offset;
	unsigned char end;
	unsigned long *pivots;
	struct maple_enode *entry;
	struct maple_node *node;
	void __rcu **slots;

	mt = mte_node_type(mas->node);
	node = mas_mn(mas);
	slots = ma_slots(node, mt);
	pivots = ma_pivots(node, mt);
	end = ma_data_end(node, mt, pivots, mas->max);
	for (offset = mas->offset; offset <= end; offset++) {
		entry = mas_slot_locked(mas, slots, offset);
		if (mte_parent(entry) == node) {
			*child = *mas;
			mas->offset = offset + 1;
			child->offset = offset;
			mas_descend(child);
			child->offset = 0;
			return true;
		}
	}
	return false;
}

/*
 * mab_shift_right() - Shift the data in mab right. Note, does not clean out the
 * old data or set b_node->b_end.
 * @b_node: the maple_big_node
 * @shift: the shift count
 */
static inline void mab_shift_right(struct maple_big_node *b_node,
				 unsigned char shift)
{
	unsigned long size = b_node->b_end * sizeof(unsigned long);

	memmove(b_node->pivot + shift, b_node->pivot, size);
	memmove(b_node->slot + shift, b_node->slot, size);
	if (b_node->type == maple_arange_64)
		memmove(b_node->gap + shift, b_node->gap, size);
}

/*
 * mab_middle_node() - Check if a middle node is needed (unlikely)
 * @b_node: the maple_big_node that contains the data.
 * @split: the potential split location
 * @slot_count: the size that can be stored in a single node being considered.
 *
 * Return: true if a middle node is required.
 */
static inline bool mab_middle_node(struct maple_big_node *b_node, int split,
				   unsigned char slot_count)
{
	unsigned char size = b_node->b_end;

	if (size >= 2 * slot_count)
		return true;

	if (!b_node->slot[split] && (size >= 2 * slot_count - 1))
		return true;

	return false;
}

/*
 * mab_no_null_split() - ensure the split doesn't fall on a NULL
 * @b_node: the maple_big_node with the data
 * @split: the suggested split location
 * @slot_count: the number of slots in the node being considered.
 *
 * Return: the split location.
 */
static inline int mab_no_null_split(struct maple_big_node *b_node,
				    unsigned char split, unsigned char slot_count)
{
	if (!b_node->slot[split]) {
		/*
		 * If the split is less than the max slot && the right side will
		 * still be sufficient, then increment the split on NULL.
		 */
		if ((split < slot_count - 1) &&
		    (b_node->b_end - split) > (mt_min_slots[b_node->type]))
			split++;
		else
			split--;
	}
	return split;
}

/*
 * mab_calc_split() - Calculate the split location and if there needs to be two
 * splits.
 * @mas: The maple state
 * @bn: The maple_big_node with the data
 * @mid_split: The second split, if required.  0 otherwise.
 *
 * Return: The first split location.  The middle split is set in @mid_split.
 */
static inline int mab_calc_split(struct ma_state *mas,
	 struct maple_big_node *bn, unsigned char *mid_split)
{
	unsigned char b_end = bn->b_end;
	int split = b_end / 2; /* Assume equal split. */
	unsigned char slot_count = mt_slots[bn->type];

	/*
	 * To support gap tracking, all NULL entries are kept together and a node cannot
	 * end on a NULL entry, with the exception of the left-most leaf.  The
	 * limitation means that the split of a node must be checked for this condition
	 * and be able to put more data in one direction or the other.
	 *
	 * Although extremely rare, it is possible to enter what is known as the 3-way
	 * split scenario.  The 3-way split comes about by means of a store of a range
	 * that overwrites the end and beginning of two full nodes.  The result is a set
	 * of entries that cannot be stored in 2 nodes.  Sometimes, these two nodes can
	 * also be located in different parent nodes which are also full.  This can
	 * carry upwards all the way to the root in the worst case.
	 */
	if (unlikely(mab_middle_node(bn, split, slot_count))) {
		split = b_end / 3;
		*mid_split = split * 2;
	} else {
		*mid_split = 0;
	}

	/* Avoid ending a node on a NULL entry */
	split = mab_no_null_split(bn, split, slot_count);

	if (unlikely(*mid_split))
		*mid_split = mab_no_null_split(bn, *mid_split, slot_count);

	return split;
}

/*
 * mas_mab_cp() - Copy data from a maple state inclusively to a maple_big_node
 * and set @b_node->b_end to the next free slot.
 * @mas: The maple state
 * @mas_start: The starting slot to copy
 * @mas_end: The end slot to copy (inclusively)
 * @b_node: The maple_big_node to place the data
 * @mab_start: The starting location in maple_big_node to store the data.
 */
static inline void mas_mab_cp(struct ma_state *mas, unsigned char mas_start,
			unsigned char mas_end, struct maple_big_node *b_node,
			unsigned char mab_start)
{
	enum maple_type mt;
	struct maple_node *node;
	void __rcu **slots;
	unsigned long *pivots, *gaps;
	int i = mas_start, j = mab_start;
	unsigned char piv_end;

	node = mas_mn(mas);
	mt = mte_node_type(mas->node);
	pivots = ma_pivots(node, mt);
	if (!i) {
		b_node->pivot[j] = pivots[i++];
		if (unlikely(i > mas_end))
			goto complete;
		j++;
	}

	piv_end = min(mas_end, mt_pivots[mt]);
	for (; i < piv_end; i++, j++) {
		b_node->pivot[j] = pivots[i];
		if (unlikely(!b_node->pivot[j]))
			goto complete;

		if (unlikely(mas->max == b_node->pivot[j]))
			goto complete;
	}

	b_node->pivot[j] = mas_safe_pivot(mas, pivots, i, mt);

complete:
	b_node->b_end = ++j;
	j -= mab_start;
	slots = ma_slots(node, mt);
	memcpy(b_node->slot + mab_start, slots + mas_start, sizeof(void *) * j);
	if (!ma_is_leaf(mt) && mt_is_alloc(mas->tree)) {
		gaps = ma_gaps(node, mt);
		memcpy(b_node->gap + mab_start, gaps + mas_start,
		       sizeof(unsigned long) * j);
	}
}

/*
 * mas_leaf_set_meta() - Set the metadata of a leaf if possible.
 * @node: The maple node
 * @mt: The maple type
 * @end: The node end
 */
static inline void mas_leaf_set_meta(struct maple_node *node,
		enum maple_type mt, unsigned char end)
{
	if (end < mt_slots[mt] - 1)
		ma_set_meta(node, mt, 0, end);
}

/*
 * mab_mas_cp() - Copy data from maple_big_node to a maple encoded node.
 * @b_node: the maple_big_node that has the data
 * @mab_start: the start location in @b_node.
 * @mab_end: The end location in @b_node (inclusively)
 * @mas: The maple state with the maple encoded node.
 */
static inline void mab_mas_cp(struct maple_big_node *b_node,
			      unsigned char mab_start, unsigned char mab_end,
			      struct ma_state *mas, bool new_max)
{
	int i, j = 0;
	enum maple_type mt = mte_node_type(mas->node);
	struct maple_node *node = mte_to_node(mas->node);
	void __rcu **slots = ma_slots(node, mt);
	unsigned long *pivots = ma_pivots(node, mt);
	unsigned long *gaps = NULL;
	unsigned char end;

	if (mab_end - mab_start > mt_pivots[mt])
		mab_end--;

	if (!pivots[mt_pivots[mt] - 1])
		slots[mt_pivots[mt]] = NULL;

	i = mab_start;
	do {
		pivots[j++] = b_node->pivot[i++];
	} while (i <= mab_end && likely(b_node->pivot[i]));

	memcpy(slots, b_node->slot + mab_start,
	       sizeof(void *) * (i - mab_start));

	if (new_max)
		mas->max = b_node->pivot[i - 1];

	end = j - 1;
	if (likely(!ma_is_leaf(mt) && mt_is_alloc(mas->tree))) {
		unsigned long max_gap = 0;
		unsigned char offset = 0;

		gaps = ma_gaps(node, mt);
		do {
			gaps[--j] = b_node->gap[--i];
			if (gaps[j] > max_gap) {
				offset = j;
				max_gap = gaps[j];
			}
		} while (j);

		ma_set_meta(node, mt, offset, end);
	} else {
		mas_leaf_set_meta(node, mt, end);
	}
}

/*
 * mas_store_b_node() - Store an @entry into the b_node while also copying the
 * data from a maple encoded node.
 * @wr_mas: the maple write state
 * @b_node: the maple_big_node to fill with data
 * @offset_end: the offset to end copying
 *
 * Return: The actual end of the data stored in @b_node
 */
static noinline_for_kasan void mas_store_b_node(struct ma_wr_state *wr_mas,
		struct maple_big_node *b_node, unsigned char offset_end)
{
	unsigned char slot;
	unsigned char b_end;
	/* Possible underflow of piv will wrap back to 0 before use. */
	unsigned long piv;
	struct ma_state *mas = wr_mas->mas;

	b_node->type = wr_mas->type;
	b_end = 0;
	slot = mas->offset;
	if (slot) {
		/* Copy start data up to insert. */
		mas_mab_cp(mas, 0, slot - 1, b_node, 0);
		b_end = b_node->b_end;
		piv = b_node->pivot[b_end - 1];
	} else
		piv = mas->min - 1;

	if (piv + 1 < mas->index) {
		/* Handle range starting after old range */
		b_node->slot[b_end] = wr_mas->content;
		if (!wr_mas->content)
			b_node->gap[b_end] = mas->index - 1 - piv;
		b_node->pivot[b_end++] = mas->index - 1;
	}

	/* Store the new entry. */
	mas->offset = b_end;
	b_node->slot[b_end] = wr_mas->entry;
	b_node->pivot[b_end] = mas->last;

	/* Appended. */
	if (mas->last >= mas->max)
		goto b_end;

	/* Handle new range ending before old range ends */
	piv = mas_safe_pivot(mas, wr_mas->pivots, offset_end, wr_mas->type);
	if (piv > mas->last) {
		if (offset_end != slot)
			wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
							  offset_end);

		b_node->slot[++b_end] = wr_mas->content;
		if (!wr_mas->content)
			b_node->gap[b_end] = piv - mas->last + 1;
		b_node->pivot[b_end] = piv;
	}

	slot = offset_end + 1;
	if (slot > mas->end)
		goto b_end;

	/* Copy end data to the end of the node. */
	mas_mab_cp(mas, slot, mas->end + 1, b_node, ++b_end);
	b_node->b_end--;
	return;

b_end:
	b_node->b_end = b_end;
}

/*
 * mas_prev_sibling() - Find the previous node with the same parent.
 * @mas: the maple state
 *
 * Return: True if there is a previous sibling, false otherwise.
 */
static inline bool mas_prev_sibling(struct ma_state *mas)
{
	unsigned int p_slot = mte_parent_slot(mas->node);

	/* For root node, p_slot is set to 0 by mte_parent_slot(). */
	if (!p_slot)
		return false;

	mas_ascend(mas);
	mas->offset = p_slot - 1;
	mas_descend(mas);
	return true;
}

/*
 * mas_next_sibling() - Find the next node with the same parent.
 * @mas: the maple state
 *
 * Return: true if there is a next sibling, false otherwise.
 */
static inline bool mas_next_sibling(struct ma_state *mas)
{
	MA_STATE(parent, mas->tree, mas->index, mas->last);

	if (mte_is_root(mas->node))
		return false;

	parent = *mas;
	mas_ascend(&parent);
	parent.offset = mte_parent_slot(mas->node) + 1;
	if (parent.offset > mas_data_end(&parent))
		return false;

	*mas = parent;
	mas_descend(mas);
	return true;
}

/*
 * mas_node_or_none() - Set the enode and state.
 * @mas: the maple state
 * @enode: The encoded maple node.
 *
 * Set the node to the enode and the status.
 */
static inline void mas_node_or_none(struct ma_state *mas,
		struct maple_enode *enode)
{
	if (enode) {
		mas->node = enode;
		mas->status = ma_active;
	} else {
		mas->node = NULL;
		mas->status = ma_none;
	}
}

/*
 * mas_wr_node_walk() - Find the correct offset for the index in the @mas.
 *                      If @mas->index cannot be found within the containing
 *                      node, we traverse to the last entry in the node.
 * @wr_mas: The maple write state
 *
 * Uses mas_slot_locked() and does not need to worry about dead nodes.
 */
static inline void mas_wr_node_walk(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char count, offset;

	if (unlikely(ma_is_dense(wr_mas->type))) {
		wr_mas->r_max = wr_mas->r_min = mas->index;
		mas->offset = mas->index = mas->min;
		return;
	}

	wr_mas->node = mas_mn(wr_mas->mas);
	wr_mas->pivots = ma_pivots(wr_mas->node, wr_mas->type);
	count = mas->end = ma_data_end(wr_mas->node, wr_mas->type,
				       wr_mas->pivots, mas->max);
	offset = mas->offset;

	while (offset < count && mas->index > wr_mas->pivots[offset])
		offset++;

	wr_mas->r_max = offset < count ? wr_mas->pivots[offset] : mas->max;
	wr_mas->r_min = mas_safe_min(mas, wr_mas->pivots, offset);
	wr_mas->offset_end = mas->offset = offset;
}

/*
 * mast_rebalance_next() - Rebalance against the next node
 * @mast: The maple subtree state
 */
static inline void mast_rebalance_next(struct maple_subtree_state *mast)
{
	unsigned char b_end = mast->bn->b_end;

	mas_mab_cp(mast->orig_r, 0, mt_slot_count(mast->orig_r->node),
		   mast->bn, b_end);
	mast->orig_r->last = mast->orig_r->max;
}

/*
 * mast_rebalance_prev() - Rebalance against the previous node
 * @mast: The maple subtree state
 */
static inline void mast_rebalance_prev(struct maple_subtree_state *mast)
{
	unsigned char end = mas_data_end(mast->orig_l) + 1;
	unsigned char b_end = mast->bn->b_end;

	mab_shift_right(mast->bn, end);
	mas_mab_cp(mast->orig_l, 0, end - 1, mast->bn, 0);
	mast->l->min = mast->orig_l->min;
	mast->orig_l->index = mast->orig_l->min;
	mast->bn->b_end = end + b_end;
	mast->l->offset += end;
}

/*
 * mast_spanning_rebalance() - Rebalance nodes with nearest neighbour favouring
 * the node to the right.  Checking the nodes to the right then the left at each
 * level upwards until root is reached.
 * Data is copied into the @mast->bn.
 * @mast: The maple_subtree_state.
 */
static inline
bool mast_spanning_rebalance(struct maple_subtree_state *mast)
{
	struct ma_state r_tmp = *mast->orig_r;
	struct ma_state l_tmp = *mast->orig_l;
	unsigned char depth = 0;

	do {
		mas_ascend(mast->orig_r);
		mas_ascend(mast->orig_l);
		depth++;
		if (mast->orig_r->offset < mas_data_end(mast->orig_r)) {
			mast->orig_r->offset++;
			do {
				mas_descend(mast->orig_r);
				mast->orig_r->offset = 0;
			} while (--depth);

			mast_rebalance_next(mast);
			*mast->orig_l = l_tmp;
			return true;
		} else if (mast->orig_l->offset != 0) {
			mast->orig_l->offset--;
			do {
				mas_descend(mast->orig_l);
				mast->orig_l->offset =
					mas_data_end(mast->orig_l);
			} while (--depth);

			mast_rebalance_prev(mast);
			*mast->orig_r = r_tmp;
			return true;
		}
	} while (!mte_is_root(mast->orig_r->node));

	*mast->orig_r = r_tmp;
	*mast->orig_l = l_tmp;
	return false;
}

/*
 * mast_ascend() - Ascend the original left and right maple states.
 * @mast: the maple subtree state.
 *
 * Ascend the original left and right sides.  Set the offsets to point to the
 * data already in the new tree (@mast->l and @mast->r).
 */
static inline void mast_ascend(struct maple_subtree_state *mast)
{
	MA_WR_STATE(wr_mas, mast->orig_r,  NULL);
	mas_ascend(mast->orig_l);
	mas_ascend(mast->orig_r);

	mast->orig_r->offset = 0;
	mast->orig_r->index = mast->r->max;
	/* last should be larger than or equal to index */
	if (mast->orig_r->last < mast->orig_r->index)
		mast->orig_r->last = mast->orig_r->index;

	wr_mas.type = mte_node_type(mast->orig_r->node);
	mas_wr_node_walk(&wr_mas);
	/* Set up the left side of things */
	mast->orig_l->offset = 0;
	mast->orig_l->index = mast->l->min;
	wr_mas.mas = mast->orig_l;
	wr_mas.type = mte_node_type(mast->orig_l->node);
	mas_wr_node_walk(&wr_mas);

	mast->bn->type = wr_mas.type;
}

/*
 * mas_new_ma_node() - Create and return a new maple node.  Helper function.
 * @mas: the maple state with the allocations.
 * @b_node: the maple_big_node with the type encoding.
 *
 * Use the node type from the maple_big_node to allocate a new node from the
 * ma_state.  This function exists mainly for code readability.
 *
 * Return: A new maple encoded node
 */
static inline struct maple_enode
*mas_new_ma_node(struct ma_state *mas, struct maple_big_node *b_node)
{
	return mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)), b_node->type);
}

/*
 * mas_mab_to_node() - Set up right and middle nodes
 *
 * @mas: the maple state that contains the allocations.
 * @b_node: the node which contains the data.
 * @left: The pointer which will have the left node
 * @right: The pointer which may have the right node
 * @middle: the pointer which may have the middle node (rare)
 * @mid_split: the split location for the middle node
 *
 * Return: the split of left.
 */
static inline unsigned char mas_mab_to_node(struct ma_state *mas,
	struct maple_big_node *b_node, struct maple_enode **left,
	struct maple_enode **right, struct maple_enode **middle,
	unsigned char *mid_split)
{
	unsigned char split = 0;
	unsigned char slot_count = mt_slots[b_node->type];

	*left = mas_new_ma_node(mas, b_node);
	*right = NULL;
	*middle = NULL;
	*mid_split = 0;

	if (b_node->b_end < slot_count) {
		split = b_node->b_end;
	} else {
		split = mab_calc_split(mas, b_node, mid_split);
		*right = mas_new_ma_node(mas, b_node);
	}

	if (*mid_split)
		*middle = mas_new_ma_node(mas, b_node);

	return split;

}

/*
 * mab_set_b_end() - Add entry to b_node at b_node->b_end and increment the end
 * pointer.
 * @b_node: the big node to add the entry
 * @mas: the maple state to get the pivot (mas->max)
 * @entry: the entry to add, if NULL nothing happens.
 */
static inline void mab_set_b_end(struct maple_big_node *b_node,
				 struct ma_state *mas,
				 void *entry)
{
	if (!entry)
		return;

	b_node->slot[b_node->b_end] = entry;
	if (mt_is_alloc(mas->tree))
		b_node->gap[b_node->b_end] = mas_max_gap(mas);
	b_node->pivot[b_node->b_end++] = mas->max;
}

/*
 * mas_set_split_parent() - combine_then_separate helper function.  Sets the parent
 * of @mas->node to either @left or @right, depending on @slot and @split
 *
 * @mas: the maple state with the node that needs a parent
 * @left: possible parent 1
 * @right: possible parent 2
 * @slot: the slot the mas->node was placed
 * @split: the split location between @left and @right
 */
static inline void mas_set_split_parent(struct ma_state *mas,
					struct maple_enode *left,
					struct maple_enode *right,
					unsigned char *slot, unsigned char split)
{
	if (mas_is_none(mas))
		return;

	if ((*slot) <= split)
		mas_set_parent(mas, mas->node, left, *slot);
	else if (right)
		mas_set_parent(mas, mas->node, right, (*slot) - split - 1);

	(*slot)++;
}

/*
 * mte_mid_split_check() - Check if the next node passes the mid-split
 * @l: Pointer to left encoded maple node.
 * @m: Pointer to middle encoded maple node.
 * @r: Pointer to right encoded maple node.
 * @slot: The offset
 * @split: The split location.
 * @mid_split: The middle split.
 */
static inline void mte_mid_split_check(struct maple_enode **l,
				       struct maple_enode **r,
				       struct maple_enode *right,
				       unsigned char slot,
				       unsigned char *split,
				       unsigned char mid_split)
{
	if (*r == right)
		return;

	if (slot < mid_split)
		return;

	*l = *r;
	*r = right;
	*split = mid_split;
}

/*
 * mast_set_split_parents() - Helper function to set three nodes parents.  Slot
 * is taken from @mast->l.
 * @mast: the maple subtree state
 * @left: the left node
 * @right: the right node
 * @split: the split location.
 */
static inline void mast_set_split_parents(struct maple_subtree_state *mast,
					  struct maple_enode *left,
					  struct maple_enode *middle,
					  struct maple_enode *right,
					  unsigned char split,
					  unsigned char mid_split)
{
	unsigned char slot;
	struct maple_enode *l = left;
	struct maple_enode *r = right;

	if (mas_is_none(mast->l))
		return;

	if (middle)
		r = middle;

	slot = mast->l->offset;

	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	mas_set_split_parent(mast->l, l, r, &slot, split);

	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	mas_set_split_parent(mast->m, l, r, &slot, split);

	mte_mid_split_check(&l, &r, right, slot, &split, mid_split);
	mas_set_split_parent(mast->r, l, r, &slot, split);
}

/*
 * mas_topiary_node() - Dispose of a single node
 * @mas: The maple state for pushing nodes
 * @in_rcu: If the tree is in rcu mode
 *
 * The node will either be RCU freed or pushed back on the maple state.
 */
static inline void mas_topiary_node(struct ma_state *mas,
		struct ma_state *tmp_mas, bool in_rcu)
{
	struct maple_node *tmp;
	struct maple_enode *enode;

	if (mas_is_none(tmp_mas))
		return;

	enode = tmp_mas->node;
	tmp = mte_to_node(enode);
	mte_set_node_dead(enode);
	ma_free_rcu(tmp);
}

/*
 * mas_topiary_replace() - Replace the data with new data, then repair the
 * parent links within the new tree.  Iterate over the dead sub-tree and collect
 * the dead subtrees and topiary the nodes that are no longer of use.
 *
 * The new tree will have up to three children with the correct parent.  Keep
 * track of the new entries as they need to be followed to find the next level
 * of new entries.
 *
 * The old tree will have up to three children with the old parent.  Keep track
 * of the old entries as they may have more nodes below replaced.  Nodes within
 * [index, last] are dead subtrees, others need to be freed and followed.
 *
 * @mas: The maple state pointing at the new data
 * @old_enode: The maple encoded node being replaced
 * @new_height: The new height of the tree as a result of the operation
 *
 */
static inline void mas_topiary_replace(struct ma_state *mas,
		struct maple_enode *old_enode, unsigned char new_height)
{
	struct ma_state tmp[3], tmp_next[3];
	MA_TOPIARY(subtrees, mas->tree);
	bool in_rcu;
	int i, n;

	/* Place data in tree & then mark node as old */
	mas_put_in_tree(mas, old_enode, new_height);

	/* Update the parent pointers in the tree */
	tmp[0] = *mas;
	tmp[0].offset = 0;
	tmp[1].status = ma_none;
	tmp[2].status = ma_none;
	while (!mte_is_leaf(tmp[0].node)) {
		n = 0;
		for (i = 0; i < 3; i++) {
			if (mas_is_none(&tmp[i]))
				continue;

			while (n < 3) {
				if (!mas_find_child(&tmp[i], &tmp_next[n]))
					break;
				n++;
			}

			mas_adopt_children(&tmp[i], tmp[i].node);
		}

		if (MAS_WARN_ON(mas, n == 0))
			break;

		while (n < 3)
			tmp_next[n++].status = ma_none;

		for (i = 0; i < 3; i++)
			tmp[i] = tmp_next[i];
	}

	/* Collect the old nodes that need to be discarded */
	if (mte_is_leaf(old_enode))
		return mas_free(mas, old_enode);

	tmp[0] = *mas;
	tmp[0].offset = 0;
	tmp[0].node = old_enode;
	tmp[1].status = ma_none;
	tmp[2].status = ma_none;
	in_rcu = mt_in_rcu(mas->tree);
	do {
		n = 0;
		for (i = 0; i < 3; i++) {
			if (mas_is_none(&tmp[i]))
				continue;

			while (n < 3) {
				if (!mas_find_child(&tmp[i], &tmp_next[n]))
					break;

				if ((tmp_next[n].min >= tmp_next->index) &&
				    (tmp_next[n].max <= tmp_next->last)) {
					mat_add(&subtrees, tmp_next[n].node);
					tmp_next[n].status = ma_none;
				} else {
					n++;
				}
			}
		}

		if (MAS_WARN_ON(mas, n == 0))
			break;

		while (n < 3)
			tmp_next[n++].status = ma_none;

		for (i = 0; i < 3; i++) {
			mas_topiary_node(mas, &tmp[i], in_rcu);
			tmp[i] = tmp_next[i];
		}
	} while (!mte_is_leaf(tmp[0].node));

	for (i = 0; i < 3; i++)
		mas_topiary_node(mas, &tmp[i], in_rcu);

	mas_mat_destroy(mas, &subtrees);
}

/*
 * mas_wmb_replace() - Write memory barrier and replace
 * @mas: The maple state
 * @old_enode: The old maple encoded node that is being replaced.
 * @new_height: The new height of the tree as a result of the operation
 *
 * Updates gap as necessary.
 */
static inline void mas_wmb_replace(struct ma_state *mas,
		struct maple_enode *old_enode, unsigned char new_height)
{
	/* Insert the new data in the tree */
	mas_topiary_replace(mas, old_enode, new_height);

	if (mte_is_leaf(mas->node))
		return;

	mas_update_gap(mas);
}

/*
 * mast_cp_to_nodes() - Copy data out to nodes.
 * @mast: The maple subtree state
 * @left: The left encoded maple node
 * @middle: The middle encoded maple node
 * @right: The right encoded maple node
 * @split: The location to split between left and (middle ? middle : right)
 * @mid_split: The location to split between middle and right.
 */
static inline void mast_cp_to_nodes(struct maple_subtree_state *mast,
	struct maple_enode *left, struct maple_enode *middle,
	struct maple_enode *right, unsigned char split, unsigned char mid_split)
{
	bool new_lmax = true;

	mas_node_or_none(mast->l, left);
	mas_node_or_none(mast->m, middle);
	mas_node_or_none(mast->r, right);

	mast->l->min = mast->orig_l->min;
	if (split == mast->bn->b_end) {
		mast->l->max = mast->orig_r->max;
		new_lmax = false;
	}

	mab_mas_cp(mast->bn, 0, split, mast->l, new_lmax);

	if (middle) {
		mab_mas_cp(mast->bn, 1 + split, mid_split, mast->m, true);
		mast->m->min = mast->bn->pivot[split] + 1;
		split = mid_split;
	}

	mast->r->max = mast->orig_r->max;
	if (right) {
		mab_mas_cp(mast->bn, 1 + split, mast->bn->b_end, mast->r, false);
		mast->r->min = mast->bn->pivot[split] + 1;
	}
}

/*
 * mast_combine_cp_left - Copy in the original left side of the tree into the
 * combined data set in the maple subtree state big node.
 * @mast: The maple subtree state
 */
static inline void mast_combine_cp_left(struct maple_subtree_state *mast)
{
	unsigned char l_slot = mast->orig_l->offset;

	if (!l_slot)
		return;

	mas_mab_cp(mast->orig_l, 0, l_slot - 1, mast->bn, 0);
}

/*
 * mast_combine_cp_right: Copy in the original right side of the tree into the
 * combined data set in the maple subtree state big node.
 * @mast: The maple subtree state
 */
static inline void mast_combine_cp_right(struct maple_subtree_state *mast)
{
	if (mast->bn->pivot[mast->bn->b_end - 1] >= mast->orig_r->max)
		return;

	mas_mab_cp(mast->orig_r, mast->orig_r->offset + 1,
		   mt_slot_count(mast->orig_r->node), mast->bn,
		   mast->bn->b_end);
	mast->orig_r->last = mast->orig_r->max;
}

/*
 * mast_sufficient: Check if the maple subtree state has enough data in the big
 * node to create at least one sufficient node
 * @mast: the maple subtree state
 */
static inline bool mast_sufficient(struct maple_subtree_state *mast)
{
	if (mast->bn->b_end > mt_min_slot_count(mast->orig_l->node))
		return true;

	return false;
}

/*
 * mast_overflow: Check if there is too much data in the subtree state for a
 * single node.
 * @mast: The maple subtree state
 */
static inline bool mast_overflow(struct maple_subtree_state *mast)
{
	if (mast->bn->b_end > mt_slot_count(mast->orig_l->node))
		return true;

	return false;
}

static inline void *mtree_range_walk(struct ma_state *mas)
{
	unsigned long *pivots;
	unsigned char offset;
	struct maple_node *node;
	struct maple_enode *next, *last;
	enum maple_type type;
	void __rcu **slots;
	unsigned char end;
	unsigned long max, min;
	unsigned long prev_max, prev_min;

	next = mas->node;
	min = mas->min;
	max = mas->max;
	do {
		last = next;
		node = mte_to_node(next);
		type = mte_node_type(next);
		pivots = ma_pivots(node, type);
		end = ma_data_end(node, type, pivots, max);
		prev_min = min;
		prev_max = max;
		if (pivots[0] >= mas->index) {
			offset = 0;
			max = pivots[0];
			goto next;
		}

		offset = 1;
		while (offset < end) {
			if (pivots[offset] >= mas->index) {
				max = pivots[offset];
				break;
			}
			offset++;
		}

		min = pivots[offset - 1] + 1;
next:
		slots = ma_slots(node, type);
		next = mt_slot(mas->tree, slots, offset);
		if (unlikely(ma_dead_node(node)))
			goto dead_node;
	} while (!ma_is_leaf(type));

	mas->end = end;
	mas->offset = offset;
	mas->index = min;
	mas->last = max;
	mas->min = prev_min;
	mas->max = prev_max;
	mas->node = last;
	return (void *)next;

dead_node:
	mas_reset(mas);
	return NULL;
}

/*
 * mas_spanning_rebalance() - Rebalance across two nodes which may not be peers.
 * @mas: The starting maple state
 * @mast: The maple_subtree_state, keeps track of 4 maple states.
 * @count: The estimated count of iterations needed.
 *
 * Follow the tree upwards from @l_mas and @r_mas for @count, or until the root
 * is hit.  First @b_node is split into two entries which are inserted into the
 * next iteration of the loop.  @b_node is returned populated with the final
 * iteration. @mas is used to obtain allocations.  orig_l_mas keeps track of the
 * nodes that will remain active by using orig_l_mas->index and orig_l_mas->last
 * to account of what has been copied into the new sub-tree.  The update of
 * orig_l_mas->last is used in mas_consume to find the slots that will need to
 * be either freed or destroyed.  orig_l_mas->depth keeps track of the height of
 * the new sub-tree in case the sub-tree becomes the full tree.
 */
static void mas_spanning_rebalance(struct ma_state *mas,
		struct maple_subtree_state *mast, unsigned char count)
{
	unsigned char split, mid_split;
	unsigned char slot = 0;
	unsigned char new_height = 0; /* used if node is a new root */
	struct maple_enode *left = NULL, *middle = NULL, *right = NULL;
	struct maple_enode *old_enode;

	MA_STATE(l_mas, mas->tree, mas->index, mas->index);
	MA_STATE(r_mas, mas->tree, mas->index, mas->last);
	MA_STATE(m_mas, mas->tree, mas->index, mas->index);

	/*
	 * The tree needs to be rebalanced and leaves need to be kept at the same level.
	 * Rebalancing is done by use of the ``struct maple_topiary``.
	 */
	mast->l = &l_mas;
	mast->m = &m_mas;
	mast->r = &r_mas;
	l_mas.status = r_mas.status = m_mas.status = ma_none;

	/* Check if this is not root and has sufficient data.  */
	if (((mast->orig_l->min != 0) || (mast->orig_r->max != ULONG_MAX)) &&
	    unlikely(mast->bn->b_end <= mt_min_slots[mast->bn->type]))
		mast_spanning_rebalance(mast);

	/*
	 * Each level of the tree is examined and balanced, pushing data to the left or
	 * right, or rebalancing against left or right nodes is employed to avoid
	 * rippling up the tree to limit the amount of churn.  Once a new sub-section of
	 * the tree is created, there may be a mix of new and old nodes.  The old nodes
	 * will have the incorrect parent pointers and currently be in two trees: the
	 * original tree and the partially new tree.  To remedy the parent pointers in
	 * the old tree, the new data is swapped into the active tree and a walk down
	 * the tree is performed and the parent pointers are updated.
	 * See mas_topiary_replace() for more information.
	 */
	while (count--) {
		mast->bn->b_end--;
		mast->bn->type = mte_node_type(mast->orig_l->node);
		split = mas_mab_to_node(mas, mast->bn, &left, &right, &middle,
					&mid_split);
		mast_set_split_parents(mast, left, middle, right, split,
				       mid_split);
		mast_cp_to_nodes(mast, left, middle, right, split, mid_split);
		new_height++;

		/*
		 * Copy data from next level in the tree to mast->bn from next
		 * iteration
		 */
		memset(mast->bn, 0, sizeof(struct maple_big_node));
		mast->bn->type = mte_node_type(left);

		/* Root already stored in l->node. */
		if (mas_is_root_limits(mast->l))
			goto new_root;

		mast_ascend(mast);
		mast_combine_cp_left(mast);
		l_mas.offset = mast->bn->b_end;
		mab_set_b_end(mast->bn, &l_mas, left);
		mab_set_b_end(mast->bn, &m_mas, middle);
		mab_set_b_end(mast->bn, &r_mas, right);

		/* Copy anything necessary out of the right node. */
		mast_combine_cp_right(mast);
		mast->orig_l->last = mast->orig_l->max;

		if (mast_sufficient(mast)) {
			if (mast_overflow(mast))
				continue;

			if (mast->orig_l->node == mast->orig_r->node) {
			       /*
				* The data in b_node should be stored in one
				* node and in the tree
				*/
				slot = mast->l->offset;
				break;
			}

			continue;
		}

		/* May be a new root stored in mast->bn */
		if (mas_is_root_limits(mast->orig_l))
			break;

		mast_spanning_rebalance(mast);

		/* rebalancing from other nodes may require another loop. */
		if (!count)
			count++;
	}

	l_mas.node = mt_mk_node(ma_mnode_ptr(mas_pop_node(mas)),
				mte_node_type(mast->orig_l->node));

	mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, &l_mas, true);
	new_height++;
	mas_set_parent(mas, left, l_mas.node, slot);
	if (middle)
		mas_set_parent(mas, middle, l_mas.node, ++slot);

	if (right)
		mas_set_parent(mas, right, l_mas.node, ++slot);

	if (mas_is_root_limits(mast->l)) {
new_root:
		mas_mn(mast->l)->parent = ma_parent_ptr(mas_tree_parent(mas));
		while (!mte_is_root(mast->orig_l->node))
			mast_ascend(mast);
	} else {
		mas_mn(&l_mas)->parent = mas_mn(mast->orig_l)->parent;
	}

	old_enode = mast->orig_l->node;
	mas->depth = l_mas.depth;
	mas->node = l_mas.node;
	mas->min = l_mas.min;
	mas->max = l_mas.max;
	mas->offset = l_mas.offset;
	mas_wmb_replace(mas, old_enode, new_height);
	mtree_range_walk(mas);
	return;
}

/*
 * mas_rebalance() - Rebalance a given node.
 * @mas: The maple state
 * @b_node: The big maple node.
 *
 * Rebalance two nodes into a single node or two new nodes that are sufficient.
 * Continue upwards until tree is sufficient.
 */
static inline void mas_rebalance(struct ma_state *mas,
				struct maple_big_node *b_node)
{
	char empty_count = mas_mt_height(mas);
	struct maple_subtree_state mast;
	unsigned char shift, b_end = ++b_node->b_end;

	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
	MA_STATE(r_mas, mas->tree, mas->index, mas->last);

	trace_ma_op(TP_FCT, mas);

	/*
	 * Rebalancing occurs if a node is insufficient.  Data is rebalanced
	 * against the node to the right if it exists, otherwise the node to the
	 * left of this node is rebalanced against this node.  If rebalancing
	 * causes just one node to be produced instead of two, then the parent
	 * is also examined and rebalanced if it is insufficient.  Every level
	 * tries to combine the data in the same way.  If one node contains the
	 * entire range of the tree, then that node is used as a new root node.
	 */

	mast.orig_l = &l_mas;
	mast.orig_r = &r_mas;
	mast.bn = b_node;
	mast.bn->type = mte_node_type(mas->node);

	l_mas = r_mas = *mas;

	if (mas_next_sibling(&r_mas)) {
		mas_mab_cp(&r_mas, 0, mt_slot_count(r_mas.node), b_node, b_end);
		r_mas.last = r_mas.index = r_mas.max;
	} else {
		mas_prev_sibling(&l_mas);
		shift = mas_data_end(&l_mas) + 1;
		mab_shift_right(b_node, shift);
		mas->offset += shift;
		mas_mab_cp(&l_mas, 0, shift - 1, b_node, 0);
		b_node->b_end = shift + b_end;
		l_mas.index = l_mas.last = l_mas.min;
	}

	return mas_spanning_rebalance(mas, &mast, empty_count);
}

/*
 * mas_split_final_node() - Split the final node in a subtree operation.
 * @mast: the maple subtree state
 * @mas: The maple state
 */
static inline void mas_split_final_node(struct maple_subtree_state *mast,
					struct ma_state *mas)
{
	struct maple_enode *ancestor;

	if (mte_is_root(mas->node)) {
		if (mt_is_alloc(mas->tree))
			mast->bn->type = maple_arange_64;
		else
			mast->bn->type = maple_range_64;
	}
	/*
	 * Only a single node is used here, could be root.
	 * The Big_node data should just fit in a single node.
	 */
	ancestor = mas_new_ma_node(mas, mast->bn);
	mas_set_parent(mas, mast->l->node, ancestor, mast->l->offset);
	mas_set_parent(mas, mast->r->node, ancestor, mast->r->offset);
	mte_to_node(ancestor)->parent = mas_mn(mas)->parent;

	mast->l->node = ancestor;
	mab_mas_cp(mast->bn, 0, mt_slots[mast->bn->type] - 1, mast->l, true);
	mas->offset = mast->bn->b_end - 1;
}

/*
 * mast_fill_bnode() - Copy data into the big node in the subtree state
 * @mast: The maple subtree state
 * @mas: the maple state
 * @skip: The number of entries to skip for new nodes insertion.
 */
static inline void mast_fill_bnode(struct maple_subtree_state *mast,
					 struct ma_state *mas,
					 unsigned char skip)
{
	bool cp = true;
	unsigned char split;

	memset(mast->bn, 0, sizeof(struct maple_big_node));

	if (mte_is_root(mas->node)) {
		cp = false;
	} else {
		mas_ascend(mas);
		mas->offset = mte_parent_slot(mas->node);
	}

	if (cp && mast->l->offset)
		mas_mab_cp(mas, 0, mast->l->offset - 1, mast->bn, 0);

	split = mast->bn->b_end;
	mab_set_b_end(mast->bn, mast->l, mast->l->node);
	mast->r->offset = mast->bn->b_end;
	mab_set_b_end(mast->bn, mast->r, mast->r->node);
	if (mast->bn->pivot[mast->bn->b_end - 1] == mas->max)
		cp = false;

	if (cp)
		mas_mab_cp(mas, split + skip, mt_slot_count(mas->node) - 1,
			   mast->bn, mast->bn->b_end);

	mast->bn->b_end--;
	mast->bn->type = mte_node_type(mas->node);
}

/*
 * mast_split_data() - Split the data in the subtree state big node into regular
 * nodes.
 * @mast: The maple subtree state
 * @mas: The maple state
 * @split: The location to split the big node
 */
static inline void mast_split_data(struct maple_subtree_state *mast,
	   struct ma_state *mas, unsigned char split)
{
	unsigned char p_slot;

	mab_mas_cp(mast->bn, 0, split, mast->l, true);
	mte_set_pivot(mast->r->node, 0, mast->r->max);
	mab_mas_cp(mast->bn, split + 1, mast->bn->b_end, mast->r, false);
	mast->l->offset = mte_parent_slot(mas->node);
	mast->l->max = mast->bn->pivot[split];
	mast->r->min = mast->l->max + 1;
	if (mte_is_leaf(mas->node))
		return;

	p_slot = mast->orig_l->offset;
	mas_set_split_parent(mast->orig_l, mast->l->node, mast->r->node,
			     &p_slot, split);
	mas_set_split_parent(mast->orig_r, mast->l->node, mast->r->node,
			     &p_slot, split);
}

/*
 * mas_push_data() - Instead of splitting a node, it is beneficial to push the
 * data to the right or left node if there is room.
 * @mas: The maple state
 * @mast: The maple subtree state
 * @left: Push left or not.
 *
 * Keeping the height of the tree low means faster lookups.
 *
 * Return: True if pushed, false otherwise.
 */
static inline bool mas_push_data(struct ma_state *mas,
				struct maple_subtree_state *mast, bool left)
{
	unsigned char slot_total = mast->bn->b_end;
	unsigned char end, space, split;

	MA_STATE(tmp_mas, mas->tree, mas->index, mas->last);
	tmp_mas = *mas;
	tmp_mas.depth = mast->l->depth;

	if (left && !mas_prev_sibling(&tmp_mas))
		return false;
	else if (!left && !mas_next_sibling(&tmp_mas))
		return false;

	end = mas_data_end(&tmp_mas);
	slot_total += end;
	space = 2 * mt_slot_count(mas->node) - 2;
	/* -2 instead of -1 to ensure there isn't a triple split */
	if (ma_is_leaf(mast->bn->type))
		space--;

	if (mas->max == ULONG_MAX)
		space--;

	if (slot_total >= space)
		return false;

	/* Get the data; Fill mast->bn */
	mast->bn->b_end++;
	if (left) {
		mab_shift_right(mast->bn, end + 1);
		mas_mab_cp(&tmp_mas, 0, end, mast->bn, 0);
		mast->bn->b_end = slot_total + 1;
	} else {
		mas_mab_cp(&tmp_mas, 0, end, mast->bn, mast->bn->b_end);
	}

	/* Configure mast for splitting of mast->bn */
	split = mt_slots[mast->bn->type] - 2;
	if (left) {
		/*  Switch mas to prev node  */
		*mas = tmp_mas;
		/* Start using mast->l for the left side. */
		tmp_mas.node = mast->l->node;
		*mast->l = tmp_mas;
	} else {
		tmp_mas.node = mast->r->node;
		*mast->r = tmp_mas;
		split = slot_total - split;
	}
	split = mab_no_null_split(mast->bn, split, mt_slots[mast->bn->type]);
	/* Update parent slot for split calculation. */
	if (left)
		mast->orig_l->offset += end + 1;

	mast_split_data(mast, mas, split);
	mast_fill_bnode(mast, mas, 2);
	mas_split_final_node(mast, mas);
	return true;
}

/*
 * mas_split() - Split data that is too big for one node into two.
 * @mas: The maple state
 * @b_node: The maple big node
 */
static void mas_split(struct ma_state *mas, struct maple_big_node *b_node)
{
	struct maple_subtree_state mast;
	int height = 0;
	unsigned int orig_height = mas_mt_height(mas);
	unsigned char mid_split, split = 0;
	struct maple_enode *old;

	/*
	 * Splitting is handled differently from any other B-tree; the Maple
	 * Tree splits upwards.  Splitting up means that the split operation
	 * occurs when the walk of the tree hits the leaves and not on the way
	 * down.  The reason for splitting up is that it is impossible to know
	 * how much space will be needed until the leaf is (or leaves are)
	 * reached.  Since overwriting data is allowed and a range could
	 * overwrite more than one range or result in changing one entry into 3
	 * entries, it is impossible to know if a split is required until the
	 * data is examined.
	 *
	 * Splitting is a balancing act between keeping allocations to a minimum
	 * and avoiding a 'jitter' event where a tree is expanded to make room
	 * for an entry followed by a contraction when the entry is removed.  To
	 * accomplish the balance, there are empty slots remaining in both left
	 * and right nodes after a split.
	 */
	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
	MA_STATE(r_mas, mas->tree, mas->index, mas->last);
	MA_STATE(prev_l_mas, mas->tree, mas->index, mas->last);
	MA_STATE(prev_r_mas, mas->tree, mas->index, mas->last);

	trace_ma_op(TP_FCT, mas);

	mast.l = &l_mas;
	mast.r = &r_mas;
	mast.orig_l = &prev_l_mas;
	mast.orig_r = &prev_r_mas;
	mast.bn = b_node;

	while (height++ <= orig_height) {
		if (mt_slots[b_node->type] > b_node->b_end) {
			mas_split_final_node(&mast, mas);
			break;
		}

		l_mas = r_mas = *mas;
		l_mas.node = mas_new_ma_node(mas, b_node);
		r_mas.node = mas_new_ma_node(mas, b_node);
		/*
		 * Another way that 'jitter' is avoided is to terminate a split up early if the
		 * left or right node has space to spare.  This is referred to as "pushing left"
		 * or "pushing right" and is similar to the B* tree, except the nodes left or
		 * right can rarely be reused due to RCU, but the ripple upwards is halted which
		 * is a significant savings.
		 */
		/* Try to push left. */
		if (mas_push_data(mas, &mast, true)) {
			height++;
			break;
		}
		/* Try to push right. */
		if (mas_push_data(mas, &mast, false)) {
			height++;
			break;
		}

		split = mab_calc_split(mas, b_node, &mid_split);
		mast_split_data(&mast, mas, split);
		/*
		 * Usually correct, mab_mas_cp in the above call overwrites
		 * r->max.
		 */
		mast.r->max = mas->max;
		mast_fill_bnode(&mast, mas, 1);
		prev_l_mas = *mast.l;
		prev_r_mas = *mast.r;
	}

	/* Set the original node as dead */
	old = mas->node;
	mas->node = l_mas.node;
	mas_wmb_replace(mas, old, height);
	mtree_range_walk(mas);
	return;
}

/*
 * mas_commit_b_node() - Commit the big node into the tree.
 * @wr_mas: The maple write state
 * @b_node: The maple big node
 */
static noinline_for_kasan void mas_commit_b_node(struct ma_wr_state *wr_mas,
			    struct maple_big_node *b_node)
{
	enum store_type type = wr_mas->mas->store_type;

	WARN_ON_ONCE(type != wr_rebalance && type != wr_split_store);

	if (type == wr_rebalance)
		return mas_rebalance(wr_mas->mas, b_node);

	return mas_split(wr_mas->mas, b_node);
}

/*
 * mas_root_expand() - Expand a root to a node
 * @mas: The maple state
 * @entry: The entry to store into the tree
 */
static inline void mas_root_expand(struct ma_state *mas, void *entry)
{
	void *contents = mas_root_locked(mas);
	enum maple_type type = maple_leaf_64;
	struct maple_node *node;
	void __rcu **slots;
	unsigned long *pivots;
	int slot = 0;

	node = mas_pop_node(mas);
	pivots = ma_pivots(node, type);
	slots = ma_slots(node, type);
	node->parent = ma_parent_ptr(mas_tree_parent(mas));
	mas->node = mt_mk_node(node, type);
	mas->status = ma_active;

	if (mas->index) {
		if (contents) {
			rcu_assign_pointer(slots[slot], contents);
			if (likely(mas->index > 1))
				slot++;
		}
		pivots[slot++] = mas->index - 1;
	}

	rcu_assign_pointer(slots[slot], entry);
	mas->offset = slot;
	pivots[slot] = mas->last;
	if (mas->last != ULONG_MAX)
		pivots[++slot] = ULONG_MAX;

	mt_set_height(mas->tree, 1);
	ma_set_meta(node, maple_leaf_64, 0, slot);
	/* swap the new root into the tree */
	rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));
	return;
}

/*
 * mas_store_root() - Storing value into root.
 * @mas: The maple state
 * @entry: The entry to store.
 *
 * There is no root node now and we are storing a value into the root - this
 * function either assigns the pointer or expands into a node.
 */
static inline void mas_store_root(struct ma_state *mas, void *entry)
{
	if (!entry) {
		if (!mas->index)
			rcu_assign_pointer(mas->tree->ma_root, NULL);
	} else if (likely((mas->last != 0) || (mas->index != 0)))
		mas_root_expand(mas, entry);
	else if (((unsigned long) (entry) & 3) == 2)
		mas_root_expand(mas, entry);
	else {
		rcu_assign_pointer(mas->tree->ma_root, entry);
		mas->status = ma_start;
	}
}

/*
 * mas_is_span_wr() - Check if the write needs to be treated as a write that
 * spans the node.
 * @wr_mas: The maple write state
 *
 * Spanning writes are writes that start in one node and end in another OR if
 * the write of a %NULL will cause the node to end with a %NULL.
 *
 * Return: True if this is a spanning write, false otherwise.
 */
static bool mas_is_span_wr(struct ma_wr_state *wr_mas)
{
	unsigned long max = wr_mas->r_max;
	unsigned long last = wr_mas->mas->last;
	enum maple_type type = wr_mas->type;
	void *entry = wr_mas->entry;

	/* Contained in this pivot, fast path */
	if (last < max)
		return false;

	if (ma_is_leaf(type)) {
		max = wr_mas->mas->max;
		if (last < max)
			return false;
	}

	if (last == max) {
		/*
		 * The last entry of leaf node cannot be NULL unless it is the
		 * rightmost node (writing ULONG_MAX), otherwise it spans slots.
		 */
		if (entry || last == ULONG_MAX)
			return false;
	}

	trace_ma_write(TP_FCT, wr_mas->mas, wr_mas->r_max, entry);
	return true;
}

static inline void mas_wr_walk_descend(struct ma_wr_state *wr_mas)
{
	wr_mas->type = mte_node_type(wr_mas->mas->node);
	mas_wr_node_walk(wr_mas);
	wr_mas->slots = ma_slots(wr_mas->node, wr_mas->type);
}

static inline void mas_wr_walk_traverse(struct ma_wr_state *wr_mas)
{
	wr_mas->mas->max = wr_mas->r_max;
	wr_mas->mas->min = wr_mas->r_min;
	wr_mas->mas->node = wr_mas->content;
	wr_mas->mas->offset = 0;
	wr_mas->mas->depth++;
}
/*
 * mas_wr_walk() - Walk the tree for a write.
 * @wr_mas: The maple write state
 *
 * Uses mas_slot_locked() and does not need to worry about dead nodes.
 *
 * Return: True if it's contained in a node, false on spanning write.
 */
static bool mas_wr_walk(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;

	while (true) {
		mas_wr_walk_descend(wr_mas);
		if (unlikely(mas_is_span_wr(wr_mas)))
			return false;

		wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
						  mas->offset);
		if (ma_is_leaf(wr_mas->type))
			return true;

		if (mas->end < mt_slots[wr_mas->type] - 1)
			wr_mas->vacant_height = mas->depth + 1;

		if (ma_is_root(mas_mn(mas))) {
			/* root needs more than 2 entries to be sufficient + 1 */
			if (mas->end > 2)
				wr_mas->sufficient_height = 1;
		} else if (mas->end > mt_min_slots[wr_mas->type] + 1)
			wr_mas->sufficient_height = mas->depth + 1;

		mas_wr_walk_traverse(wr_mas);
	}

	return true;
}

static void mas_wr_walk_index(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;

	while (true) {
		mas_wr_walk_descend(wr_mas);
		wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
						  mas->offset);
		if (ma_is_leaf(wr_mas->type))
			return;
		mas_wr_walk_traverse(wr_mas);
	}
}
/*
 * mas_extend_spanning_null() - Extend a store of a %NULL to include surrounding %NULLs.
 * @l_wr_mas: The left maple write state
 * @r_wr_mas: The right maple write state
 */
static inline void mas_extend_spanning_null(struct ma_wr_state *l_wr_mas,
					    struct ma_wr_state *r_wr_mas)
{
	struct ma_state *r_mas = r_wr_mas->mas;
	struct ma_state *l_mas = l_wr_mas->mas;
	unsigned char l_slot;

	l_slot = l_mas->offset;
	if (!l_wr_mas->content)
		l_mas->index = l_wr_mas->r_min;

	if ((l_mas->index == l_wr_mas->r_min) &&
		 (l_slot &&
		  !mas_slot_locked(l_mas, l_wr_mas->slots, l_slot - 1))) {
		if (l_slot > 1)
			l_mas->index = l_wr_mas->pivots[l_slot - 2] + 1;
		else
			l_mas->index = l_mas->min;

		l_mas->offset = l_slot - 1;
	}

	if (!r_wr_mas->content) {
		if (r_mas->last < r_wr_mas->r_max)
			r_mas->last = r_wr_mas->r_max;
		r_mas->offset++;
	} else if ((r_mas->last == r_wr_mas->r_max) &&
	    (r_mas->last < r_mas->max) &&
	    !mas_slot_locked(r_mas, r_wr_mas->slots, r_mas->offset + 1)) {
		r_mas->last = mas_safe_pivot(r_mas, r_wr_mas->pivots,
					     r_wr_mas->type, r_mas->offset + 1);
		r_mas->offset++;
	}
}

static inline void *mas_state_walk(struct ma_state *mas)
{
	void *entry;

	entry = mas_start(mas);
	if (mas_is_none(mas))
		return NULL;

	if (mas_is_ptr(mas))
		return entry;

	return mtree_range_walk(mas);
}

/*
 * mtree_lookup_walk() - Internal quick lookup that does not keep maple state up
 * to date.
 *
 * @mas: The maple state.
 *
 * Note: Leaves mas in undesirable state.
 * Return: The entry for @mas->index or %NULL on dead node.
 */
static inline void *mtree_lookup_walk(struct ma_state *mas)
{
	unsigned long *pivots;
	unsigned char offset;
	struct maple_node *node;
	struct maple_enode *next;
	enum maple_type type;
	void __rcu **slots;
	unsigned char end;

	next = mas->node;
	do {
		node = mte_to_node(next);
		type = mte_node_type(next);
		pivots = ma_pivots(node, type);
		end = mt_pivots[type];
		offset = 0;
		do {
			if (pivots[offset] >= mas->index)
				break;
		} while (++offset < end);

		slots = ma_slots(node, type);
		next = mt_slot(mas->tree, slots, offset);
		if (unlikely(ma_dead_node(node)))
			goto dead_node;
	} while (!ma_is_leaf(type));

	return (void *)next;

dead_node:
	mas_reset(mas);
	return NULL;
}

static void mte_destroy_walk(struct maple_enode *, struct maple_tree *);
/*
 * mas_new_root() - Create a new root node that only contains the entry passed
 * in.
 * @mas: The maple state
 * @entry: The entry to store.
 *
 * Only valid when the index == 0 and the last == ULONG_MAX
 */
static inline void mas_new_root(struct ma_state *mas, void *entry)
{
	struct maple_enode *root = mas_root_locked(mas);
	enum maple_type type = maple_leaf_64;
	struct maple_node *node;
	void __rcu **slots;
	unsigned long *pivots;

	WARN_ON_ONCE(mas->index || mas->last != ULONG_MAX);

	if (!entry) {
		mt_set_height(mas->tree, 0);
		rcu_assign_pointer(mas->tree->ma_root, entry);
		mas->status = ma_start;
		goto done;
	}

	node = mas_pop_node(mas);
	pivots = ma_pivots(node, type);
	slots = ma_slots(node, type);
	node->parent = ma_parent_ptr(mas_tree_parent(mas));
	mas->node = mt_mk_node(node, type);
	mas->status = ma_active;
	rcu_assign_pointer(slots[0], entry);
	pivots[0] = mas->last;
	mt_set_height(mas->tree, 1);
	rcu_assign_pointer(mas->tree->ma_root, mte_mk_root(mas->node));

done:
	if (xa_is_node(root))
		mte_destroy_walk(root, mas->tree);

	return;
}
/*
 * mas_wr_spanning_store() - Create a subtree with the store operation completed
 * and new nodes where necessary, then place the sub-tree in the actual tree.
 * Note that mas is expected to point to the node which caused the store to
 * span.
 * @wr_mas: The maple write state
 */
static noinline void mas_wr_spanning_store(struct ma_wr_state *wr_mas)
{
	struct maple_subtree_state mast;
	struct maple_big_node b_node;
	struct ma_state *mas;
	unsigned char height;

	/* Left and Right side of spanning store */
	MA_STATE(l_mas, NULL, 0, 0);
	MA_STATE(r_mas, NULL, 0, 0);
	MA_WR_STATE(r_wr_mas, &r_mas, wr_mas->entry);
	MA_WR_STATE(l_wr_mas, &l_mas, wr_mas->entry);

	/*
	 * A store operation that spans multiple nodes is called a spanning
	 * store and is handled early in the store call stack by the function
	 * mas_is_span_wr().  When a spanning store is identified, the maple
	 * state is duplicated.  The first maple state walks the left tree path
	 * to ``index``, the duplicate walks the right tree path to ``last``.
	 * The data in the two nodes are combined into a single node, two nodes,
	 * or possibly three nodes (see the 3-way split above).  A ``NULL``
	 * written to the last entry of a node is considered a spanning store as
	 * a rebalance is required for the operation to complete and an overflow
	 * of data may happen.
	 */
	mas = wr_mas->mas;
	trace_ma_op(TP_FCT, mas);

	if (unlikely(!mas->index && mas->last == ULONG_MAX))
		return mas_new_root(mas, wr_mas->entry);
	/*
	 * Node rebalancing may occur due to this store, so there may be three new
	 * entries per level plus a new root.
	 */
	height = mas_mt_height(mas);

	/*
	 * Set up right side.  Need to get to the next offset after the spanning
	 * store to ensure it's not NULL and to combine both the next node and
	 * the node with the start together.
	 */
	r_mas = *mas;
	/* Avoid overflow, walk to next slot in the tree. */
	if (r_mas.last + 1)
		r_mas.last++;

	r_mas.index = r_mas.last;
	mas_wr_walk_index(&r_wr_mas);
	r_mas.last = r_mas.index = mas->last;

	/* Set up left side. */
	l_mas = *mas;
	mas_wr_walk_index(&l_wr_mas);

	if (!wr_mas->entry) {
		mas_extend_spanning_null(&l_wr_mas, &r_wr_mas);
		mas->offset = l_mas.offset;
		mas->index = l_mas.index;
		mas->last = l_mas.last = r_mas.last;
	}

	/* expanding NULLs may make this cover the entire range */
	if (!l_mas.index && r_mas.last == ULONG_MAX) {
		mas_set_range(mas, 0, ULONG_MAX);
		return mas_new_root(mas, wr_mas->entry);
	}

	memset(&b_node, 0, sizeof(struct maple_big_node));
	/* Copy l_mas and store the value in b_node. */
	mas_store_b_node(&l_wr_mas, &b_node, l_mas.end);
	/* Copy r_mas into b_node if there is anything to copy. */
	if (r_mas.max > r_mas.last)
		mas_mab_cp(&r_mas, r_mas.offset, r_mas.end,
			   &b_node, b_node.b_end + 1);
	else
		b_node.b_end++;

	/* Stop spanning searches by searching for just index. */
	l_mas.index = l_mas.last = mas->index;

	mast.bn = &b_node;
	mast.orig_l = &l_mas;
	mast.orig_r = &r_mas;
	/* Combine l_mas and r_mas and split them up evenly again. */
	return mas_spanning_rebalance(mas, &mast, height + 1);
}

/*
 * mas_wr_node_store() - Attempt to store the value in a node
 * @wr_mas: The maple write state
 *
 * Attempts to reuse the node, but may allocate.
 */
static inline void mas_wr_node_store(struct ma_wr_state *wr_mas,
				     unsigned char new_end)
{
	struct ma_state *mas = wr_mas->mas;
	void __rcu **dst_slots;
	unsigned long *dst_pivots;
	unsigned char dst_offset, offset_end = wr_mas->offset_end;
	struct maple_node reuse, *newnode;
	unsigned char copy_size, node_pivots = mt_pivots[wr_mas->type];
	bool in_rcu = mt_in_rcu(mas->tree);
	unsigned char height = mas_mt_height(mas);

	if (mas->last == wr_mas->end_piv)
		offset_end++; /* don't copy this offset */

	/* set up node. */
	if (in_rcu) {
		newnode = mas_pop_node(mas);
	} else {
		memset(&reuse, 0, sizeof(struct maple_node));
		newnode = &reuse;
	}

	newnode->parent = mas_mn(mas)->parent;
	dst_pivots = ma_pivots(newnode, wr_mas->type);
	dst_slots = ma_slots(newnode, wr_mas->type);
	/* Copy from start to insert point */
	memcpy(dst_pivots, wr_mas->pivots, sizeof(unsigned long) * mas->offset);
	memcpy(dst_slots, wr_mas->slots, sizeof(void *) * mas->offset);

	/* Handle insert of new range starting after old range */
	if (wr_mas->r_min < mas->index) {
		rcu_assign_pointer(dst_slots[mas->offset], wr_mas->content);
		dst_pivots[mas->offset++] = mas->index - 1;
	}

	/* Store the new entry and range end. */
	if (mas->offset < node_pivots)
		dst_pivots[mas->offset] = mas->last;
	rcu_assign_pointer(dst_slots[mas->offset], wr_mas->entry);

	/*
	 * this range wrote to the end of the node or it overwrote the rest of
	 * the data
	 */
	if (offset_end > mas->end)
		goto done;

	dst_offset = mas->offset + 1;
	/* Copy to the end of node if necessary. */
	copy_size = mas->end - offset_end + 1;
	memcpy(dst_slots + dst_offset, wr_mas->slots + offset_end,
	       sizeof(void *) * copy_size);
	memcpy(dst_pivots + dst_offset, wr_mas->pivots + offset_end,
	       sizeof(unsigned long) * (copy_size - 1));

	if (new_end < node_pivots)
		dst_pivots[new_end] = mas->max;

done:
	mas_leaf_set_meta(newnode, maple_leaf_64, new_end);
	if (in_rcu) {
		struct maple_enode *old_enode = mas->node;

		mas->node = mt_mk_node(newnode, wr_mas->type);
		mas_replace_node(mas, old_enode, height);
	} else {
		memcpy(wr_mas->node, newnode, sizeof(struct maple_node));
	}
	trace_ma_write(TP_FCT, mas, 0, wr_mas->entry);
	mas_update_gap(mas);
	mas->end = new_end;
	return;
}

/*
 * mas_wr_slot_store: Attempt to store a value in a slot.
 * @wr_mas: the maple write state
 */
static inline void mas_wr_slot_store(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char offset = mas->offset;
	void __rcu **slots = wr_mas->slots;
	bool gap = false;

	gap |= !mt_slot_locked(mas->tree, slots, offset);
	gap |= !mt_slot_locked(mas->tree, slots, offset + 1);

	if (wr_mas->offset_end - offset == 1) {
		if (mas->index == wr_mas->r_min) {
			/* Overwriting the range and a part of the next one */
			rcu_assign_pointer(slots[offset], wr_mas->entry);
			wr_mas->pivots[offset] = mas->last;
		} else {
			/* Overwriting a part of the range and the next one */
			rcu_assign_pointer(slots[offset + 1], wr_mas->entry);
			wr_mas->pivots[offset] = mas->index - 1;
			mas->offset++; /* Keep mas accurate. */
		}
	} else {
		WARN_ON_ONCE(mt_in_rcu(mas->tree));
		/*
		 * Expand the range, only partially overwriting the previous and
		 * next ranges
		 */
		gap |= !mt_slot_locked(mas->tree, slots, offset + 2);
		rcu_assign_pointer(slots[offset + 1], wr_mas->entry);
		wr_mas->pivots[offset] = mas->index - 1;
		wr_mas->pivots[offset + 1] = mas->last;
		mas->offset++; /* Keep mas accurate. */
	}

	trace_ma_write(TP_FCT, mas, 0, wr_mas->entry);
	/*
	 * Only update gap when the new entry is empty or there is an empty
	 * entry in the original two ranges.
	 */
	if (!wr_mas->entry || gap)
		mas_update_gap(mas);

	return;
}

static inline void mas_wr_extend_null(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;

	if (!wr_mas->slots[wr_mas->offset_end]) {
		/* If this one is null, the next and prev are not */
		mas->last = wr_mas->end_piv;
	} else {
		/* Check next slot(s) if we are overwriting the end */
		if ((mas->last == wr_mas->end_piv) &&
		    (mas->end != wr_mas->offset_end) &&
		    !wr_mas->slots[wr_mas->offset_end + 1]) {
			wr_mas->offset_end++;
			if (wr_mas->offset_end == mas->end)
				mas->last = mas->max;
			else
				mas->last = wr_mas->pivots[wr_mas->offset_end];
			wr_mas->end_piv = mas->last;
		}
	}

	if (!wr_mas->content) {
		/* If this one is null, the next and prev are not */
		mas->index = wr_mas->r_min;
	} else {
		/* Check prev slot if we are overwriting the start */
		if (mas->index == wr_mas->r_min && mas->offset &&
		    !wr_mas->slots[mas->offset - 1]) {
			mas->offset--;
			wr_mas->r_min = mas->index =
				mas_safe_min(mas, wr_mas->pivots, mas->offset);
			wr_mas->r_max = wr_mas->pivots[mas->offset];
		}
	}
}

static inline void mas_wr_end_piv(struct ma_wr_state *wr_mas)
{
	while ((wr_mas->offset_end < wr_mas->mas->end) &&
	       (wr_mas->mas->last > wr_mas->pivots[wr_mas->offset_end]))
		wr_mas->offset_end++;

	if (wr_mas->offset_end < wr_mas->mas->end)
		wr_mas->end_piv = wr_mas->pivots[wr_mas->offset_end];
	else
		wr_mas->end_piv = wr_mas->mas->max;
}

static inline unsigned char mas_wr_new_end(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char new_end = mas->end + 2;

	new_end -= wr_mas->offset_end - mas->offset;
	if (wr_mas->r_min == mas->index)
		new_end--;

	if (wr_mas->end_piv == mas->last)
		new_end--;

	return new_end;
}

/*
 * mas_wr_append: Attempt to append
 * @wr_mas: the maple write state
 * @new_end: The end of the node after the modification
 *
 * This is currently unsafe in rcu mode since the end of the node may be cached
 * by readers while the node contents may be updated which could result in
 * inaccurate information.
 */
static inline void mas_wr_append(struct ma_wr_state *wr_mas,
		unsigned char new_end)
{
	struct ma_state *mas = wr_mas->mas;
	void __rcu **slots;
	unsigned char end = mas->end;

	if (new_end < mt_pivots[wr_mas->type]) {
		wr_mas->pivots[new_end] = wr_mas->pivots[end];
		ma_set_meta(wr_mas->node, wr_mas->type, 0, new_end);
	}

	slots = wr_mas->slots;
	if (new_end == end + 1) {
		if (mas->last == wr_mas->r_max) {
			/* Append to end of range */
			rcu_assign_pointer(slots[new_end], wr_mas->entry);
			wr_mas->pivots[end] = mas->index - 1;
			mas->offset = new_end;
		} else {
			/* Append to start of range */
			rcu_assign_pointer(slots[new_end], wr_mas->content);
			wr_mas->pivots[end] = mas->last;
			rcu_assign_pointer(slots[end], wr_mas->entry);
		}
	} else {
		/* Append to the range without touching any boundaries. */
		rcu_assign_pointer(slots[new_end], wr_mas->content);
		wr_mas->pivots[end + 1] = mas->last;
		rcu_assign_pointer(slots[end + 1], wr_mas->entry);
		wr_mas->pivots[end] = mas->index - 1;
		mas->offset = end + 1;
	}

	if (!wr_mas->content || !wr_mas->entry)
		mas_update_gap(mas);

	mas->end = new_end;
	trace_ma_write(TP_FCT, mas, new_end, wr_mas->entry);
	return;
}

/*
 * mas_wr_bnode() - Slow path for a modification.
 * @wr_mas: The write maple state
 *
 * This is where split, rebalance end up.
 */
static void mas_wr_bnode(struct ma_wr_state *wr_mas)
{
	struct maple_big_node b_node;

	trace_ma_write(TP_FCT, wr_mas->mas, 0, wr_mas->entry);
	memset(&b_node, 0, sizeof(struct maple_big_node));
	mas_store_b_node(wr_mas, &b_node, wr_mas->offset_end);
	mas_commit_b_node(wr_mas, &b_node);
}

/*
 * mas_wr_store_entry() - Internal call to store a value
 * @wr_mas: The maple write state
 */
static inline void mas_wr_store_entry(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char new_end = mas_wr_new_end(wr_mas);

	switch (mas->store_type) {
	case wr_exact_fit:
		rcu_assign_pointer(wr_mas->slots[mas->offset], wr_mas->entry);
		if (!!wr_mas->entry ^ !!wr_mas->content)
			mas_update_gap(mas);
		break;
	case wr_append:
		mas_wr_append(wr_mas, new_end);
		break;
	case wr_slot_store:
		mas_wr_slot_store(wr_mas);
		break;
	case wr_node_store:
		mas_wr_node_store(wr_mas, new_end);
		break;
	case wr_spanning_store:
		mas_wr_spanning_store(wr_mas);
		break;
	case wr_split_store:
	case wr_rebalance:
		mas_wr_bnode(wr_mas);
		break;
	case wr_new_root:
		mas_new_root(mas, wr_mas->entry);
		break;
	case wr_store_root:
		mas_store_root(mas, wr_mas->entry);
		break;
	case wr_invalid:
		MT_BUG_ON(mas->tree, 1);
	}

	return;
}

static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;

	if (!mas_is_active(mas)) {
		if (mas_is_start(mas))
			goto set_content;

		if (unlikely(mas_is_paused(mas)))
			goto reset;

		if (unlikely(mas_is_none(mas)))
			goto reset;

		if (unlikely(mas_is_overflow(mas)))
			goto reset;

		if (unlikely(mas_is_underflow(mas)))
			goto reset;
	}

	/*
	 * A less strict version of mas_is_span_wr() where we allow spanning
	 * writes within this node.  This is to stop partial walks in
	 * mas_prealloc() from being reset.
	 */
	if (mas->last > mas->max)
		goto reset;

	if (wr_mas->entry)
		goto set_content;

	if (mte_is_leaf(mas->node) && mas->last == mas->max)
		goto reset;

	goto set_content;

reset:
	mas_reset(mas);
set_content:
	wr_mas->content = mas_start(mas);
}

/**
 * mas_prealloc_calc() - Calculate number of nodes needed for a
 * given store oepration
 * @wr_mas: The maple write state
 * @entry: The entry to store into the tree
 *
 * Return: Number of nodes required for preallocation.
 */
static inline void mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char height = mas_mt_height(mas);
	int ret = height * 3 + 1;
	unsigned char delta = height - wr_mas->vacant_height;

	switch (mas->store_type) {
	case wr_exact_fit:
	case wr_append:
	case wr_slot_store:
		ret = 0;
		break;
	case wr_spanning_store:
		if (wr_mas->sufficient_height < wr_mas->vacant_height)
			ret = (height - wr_mas->sufficient_height) * 3 + 1;
		else
			ret = delta * 3 + 1;
		break;
	case wr_split_store:
		ret = delta * 2 + 1;
		break;
	case wr_rebalance:
		if (wr_mas->sufficient_height < wr_mas->vacant_height)
			ret = (height - wr_mas->sufficient_height) * 2 + 1;
		else
			ret = delta * 2 + 1;
		break;
	case wr_node_store:
		ret = mt_in_rcu(mas->tree) ? 1 : 0;
		break;
	case wr_new_root:
		ret = 1;
		break;
	case wr_store_root:
		if (likely((mas->last != 0) || (mas->index != 0)))
			ret = 1;
		else if (((unsigned long) (entry) & 3) == 2)
			ret = 1;
		else
			ret = 0;
		break;
	case wr_invalid:
		WARN_ON_ONCE(1);
	}

	mas->node_request = ret;
}

/*
 * mas_wr_store_type() - Determine the store type for a given
 * store operation.
 * @wr_mas: The maple write state
 *
 * Return: the type of store needed for the operation
 */
static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
{
	struct ma_state *mas = wr_mas->mas;
	unsigned char new_end;

	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
		return wr_store_root;

	if (unlikely(!mas_wr_walk(wr_mas)))
		return wr_spanning_store;

	/* At this point, we are at the leaf node that needs to be altered. */
	mas_wr_end_piv(wr_mas);
	if (!wr_mas->entry)
		mas_wr_extend_null(wr_mas);

	if ((wr_mas->r_min == mas->index) && (wr_mas->r_max == mas->last))
		return wr_exact_fit;

	if (unlikely(!mas->index && mas->last == ULONG_MAX))
		return wr_new_root;

	new_end = mas_wr_new_end(wr_mas);
	/* Potential spanning rebalance collapsing a node */
	if (new_end < mt_min_slots[wr_mas->type]) {
		if (!mte_is_root(mas->node))
			return  wr_rebalance;
		return wr_node_store;
	}

	if (new_end >= mt_slots[wr_mas->type])
		return wr_split_store;

	if (!mt_in_rcu(mas->tree) && (mas->offset == mas->end))
		return wr_append;

	if ((new_end == mas->end) && (!mt_in_rcu(mas->tree) ||
		(wr_mas->offset_end - mas->offset == 1)))
		return wr_slot_store;

	return wr_node_store;
}

/**
 * mas_wr_preallocate() - Preallocate enough nodes for a store operation
 * @wr_mas: The maple write state
 * @entry: The entry that will be stored
 *
 */
static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry)
{
	struct ma_state *mas = wr_mas->mas;

	mas_wr_prealloc_setup(wr_mas);
	mas->store_type = mas_wr_store_type(wr_mas);
	mas_prealloc_calc(wr_mas, entry);
	if (!mas->node_request)
		return;

	mas_alloc_nodes(mas, GFP_NOWAIT);
}

/**
 * mas_insert() - Internal call to insert a value
 * @mas: The maple state
 * @entry: The entry to store
 *
 * Return: %NULL or the contents that already exists at the requested index
 * otherwise.  The maple state needs to be checked for error conditions.
 */
static inline void *mas_insert(struct ma_state *mas, void *entry)
{
	MA_WR_STATE(wr_mas, mas, entry);

	/*
	 * Inserting a new range inserts either 0, 1, or 2 pivots within the
	 * tree.  If the insert fits exactly into an existing gap with a value
	 * of NULL, then the slot only needs to be written with the new value.
	 * If the range being inserted is adjacent to another range, then only a
	 * single pivot needs to be inserted (as well as writing the entry).  If
	 * the new range is within a gap but does not touch any other ranges,
	 * then two pivots need to be inserted: the start - 1, and the end.  As
	 * usual, the entry must be written.  Most operations require a new node
	 * to be allocated and replace an existing node to ensure RCU safety,
	 * when in RCU mode.  The exception to requiring a newly allocated node
	 * is when inserting at the end of a node (appending).  When done
	 * carefully, appending can reuse the node in place.
	 */
	wr_mas.content = mas_start(mas);
	if (wr_mas.content)
		goto exists;

	mas_wr_preallocate(&wr_mas, entry);
	if (mas_is_err(mas))
		return NULL;

	/* spanning writes always overwrite something */
	if (mas->store_type == wr_spanning_store)
		goto exists;

	/* At this point, we are at the leaf node that needs to be altered. */
	if (mas->store_type != wr_new_root && mas->store_type != wr_store_root) {
		wr_mas.offset_end = mas->offset;
		wr_mas.end_piv = wr_mas.r_max;

		if (wr_mas.content || (mas->last > wr_mas.r_max))
			goto exists;
	}

	mas_wr_store_entry(&wr_mas);
	return wr_mas.content;

exists:
	mas_set_err(mas, -EEXIST);
	return wr_mas.content;

}

/**
 * mas_alloc_cyclic() - Internal call to find somewhere to store an entry
 * @mas: The maple state.
 * @startp: Pointer to ID.
 * @range_lo: Lower bound of range to search.
 * @range_hi: Upper bound of range to search.
 * @entry: The entry to store.
 * @next: Pointer to next ID to allocate.
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * Return: 0 if the allocation succeeded without wrapping, 1 if the
 * allocation succeeded after wrapping, or -EBUSY if there are no
 * free entries.
 */
int mas_alloc_cyclic(struct ma_state *mas, unsigned long *startp,
		void *entry, unsigned long range_lo, unsigned long range_hi,
		unsigned long *next, gfp_t gfp)
{
	unsigned long min = range_lo;
	int ret = 0;

	range_lo = max(min, *next);
	ret = mas_empty_area(mas, range_lo, range_hi, 1);
	if ((mas->tree->ma_flags & MT_FLAGS_ALLOC_WRAPPED) && ret == 0) {
		mas->tree->ma_flags &= ~MT_FLAGS_ALLOC_WRAPPED;
		ret = 1;
	}
	if (ret < 0 && range_lo > min) {
		mas_reset(mas);
		ret = mas_empty_area(mas, min, range_hi, 1);
		if (ret == 0)
			ret = 1;
	}
	if (ret < 0)
		return ret;

	do {
		mas_insert(mas, entry);
	} while (mas_nomem(mas, gfp));
	if (mas_is_err(mas))
		return xa_err(mas->node);

	*startp = mas->index;
	*next = *startp + 1;
	if (*next == 0)
		mas->tree->ma_flags |= MT_FLAGS_ALLOC_WRAPPED;

	mas_destroy(mas);
	return ret;
}
EXPORT_SYMBOL(mas_alloc_cyclic);

static __always_inline void mas_rewalk(struct ma_state *mas, unsigned long index)
{
retry:
	mas_set(mas, index);
	mas_state_walk(mas);
	if (mas_is_start(mas))
		goto retry;
}

static __always_inline bool mas_rewalk_if_dead(struct ma_state *mas,
		struct maple_node *node, const unsigned long index)
{
	if (unlikely(ma_dead_node(node))) {
		mas_rewalk(mas, index);
		return true;
	}
	return false;
}

/*
 * mas_prev_node() - Find the prev non-null entry at the same level in the
 * tree.  The prev value will be mas->node[mas->offset] or the status will be
 * ma_none.
 * @mas: The maple state
 * @min: The lower limit to search
 *
 * The prev node value will be mas->node[mas->offset] or the status will be
 * ma_none.
 * Return: 1 if the node is dead, 0 otherwise.
 */
static int mas_prev_node(struct ma_state *mas, unsigned long min)
{
	enum maple_type mt;
	int offset, level;
	void __rcu **slots;
	struct maple_node *node;
	unsigned long *pivots;
	unsigned long max;

	node = mas_mn(mas);
	if (!mas->min)
		goto no_entry;

	max = mas->min - 1;
	if (max < min)
		goto no_entry;

	level = 0;
	do {
		if (ma_is_root(node))
			goto no_entry;

		/* Walk up. */
		if (unlikely(mas_ascend(mas)))
			return 1;
		offset = mas->offset;
		level++;
		node = mas_mn(mas);
	} while (!offset);

	offset--;
	mt = mte_node_type(mas->node);
	while (level > 1) {
		level--;
		slots = ma_slots(node, mt);
		mas->node = mas_slot(mas, slots, offset);
		if (unlikely(ma_dead_node(node)))
			return 1;

		mt = mte_node_type(mas->node);
		node = mas_mn(mas);
		pivots = ma_pivots(node, mt);
		offset = ma_data_end(node, mt, pivots, max);
		if (unlikely(ma_dead_node(node)))
			return 1;
	}

	slots = ma_slots(node, mt);
	mas->node = mas_slot(mas, slots, offset);
	pivots = ma_pivots(node, mt);
	if (unlikely(ma_dead_node(node)))
		return 1;

	if (likely(offset))
		mas->min = pivots[offset - 1] + 1;
	mas->max = max;
	mas->offset = mas_data_end(mas);
	if (unlikely(mte_dead_node(mas->node)))
		return 1;

	mas->end = mas->offset;
	return 0;

no_entry:
	if (unlikely(ma_dead_node(node)))
		return 1;

	mas->status = ma_underflow;
	return 0;
}

/*
 * mas_prev_slot() - Get the entry in the previous slot
 *
 * @mas: The maple state
 * @min: The minimum starting range
 * @empty: Can be empty
 *
 * Return: The entry in the previous slot which is possibly NULL
 */
static void *mas_prev_slot(struct ma_state *mas, unsigned long min, bool empty)
{
	void *entry;
	void __rcu **slots;
	unsigned long pivot;
	enum maple_type type;
	unsigned long *pivots;
	struct maple_node *node;
	unsigned long save_point = mas->index;

retry:
	node = mas_mn(mas);
	type = mte_node_type(mas->node);
	pivots = ma_pivots(node, type);
	if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
		goto retry;

	if (mas->min <= min) {
		pivot = mas_safe_min(mas, pivots, mas->offset);

		if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
			goto retry;

		if (pivot <= min)
			goto underflow;
	}

again:
	if (likely(mas->offset)) {
		mas->offset--;
		mas->last = mas->index - 1;
		mas->index = mas_safe_min(mas, pivots, mas->offset);
	} else  {
		if (mas->index <= min)
			goto underflow;

		if (mas_prev_node(mas, min)) {
			mas_rewalk(mas, save_point);
			goto retry;
		}

		if (WARN_ON_ONCE(mas_is_underflow(mas)))
			return NULL;

		mas->last = mas->max;
		node = mas_mn(mas);
		type = mte_node_type(mas->node);
		pivots = ma_pivots(node, type);
		mas->index = pivots[mas->offset - 1] + 1;
	}

	slots = ma_slots(node, type);
	entry = mas_slot(mas, slots, mas->offset);
	if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
		goto retry;

	if (likely(entry))
		return entry;

	if (!empty) {
		if (mas->index <= min)
			goto underflow;

		goto again;
	}

	return entry;

underflow:
	mas->status = ma_underflow;
	return NULL;
}

/*
 * mas_next_node() - Get the next node at the same level in the tree.
 * @mas: The maple state
 * @node: The maple node
 * @max: The maximum pivot value to check.
 *
 * The next value will be mas->node[mas->offset] or the status will have
 * overflowed.
 * Return: 1 on dead node, 0 otherwise.
 */
static int mas_next_node(struct ma_state *mas, struct maple_node *node,
		unsigned long max)
{
	unsigned long min;
	unsigned long *pivots;
	struct maple_enode *enode;
	struct maple_node *tmp;
	int level = 0;
	unsigned char node_end;
	enum maple_type mt;
	void __rcu **slots;

	if (mas->max >= max)
		goto overflow;

	min = mas->max + 1;
	level = 0;
	do {
		if (ma_is_root(node))
			goto overflow;

		/* Walk up. */
		if (unlikely(mas_ascend(mas)))
			return 1;

		level++;
		node = mas_mn(mas);
		mt = mte_node_type(mas->node);
		pivots = ma_pivots(node, mt);
		node_end = ma_data_end(node, mt, pivots, mas->max);
		if (unlikely(ma_dead_node(node)))
			return 1;

	} while (unlikely(mas->offset == node_end));

	slots = ma_slots(node, mt);
	mas->offset++;
	enode = mas_slot(mas, slots, mas->offset);
	if (unlikely(ma_dead_node(node)))
		return 1;

	if (level > 1)
		mas->offset = 0;

	while (unlikely(level > 1)) {
		level--;
		mas->node = enode;
		node = mas_mn(mas);
		mt = mte_node_type(mas->node);
		slots = ma_slots(node, mt);
		enode = mas_slot(mas, slots, 0);
		if (unlikely(ma_dead_node(node)))
			return 1;
	}

	if (!mas->offset)
		pivots = ma_pivots(node, mt);

	mas->max = mas_safe_pivot(mas, pivots, mas->offset, mt);
	tmp = mte_to_node(enode);
	mt = mte_node_type(enode);
	pivots = ma_pivots(tmp, mt);
	mas->end = ma_data_end(tmp, mt, pivots, mas->max);
	if (unlikely(ma_dead_node(node)))
		return 1;

	mas->node = enode;
	mas->min = min;
	return 0;

overflow:
	if (unlikely(ma_dead_node(node)))
		return 1;

	mas->status = ma_overflow;
	return 0;
}

/*
 * mas_next_slot() - Get the entry in the next slot
 *
 * @mas: The maple state
 * @max: The maximum starting range
 * @empty: Can be empty
 *
 * Return: The entry in the next slot which is possibly NULL
 */
static void *mas_next_slot(struct ma_state *mas, unsigned long max, bool empty)
{
	void __rcu **slots;
	unsigned long *pivots;
	unsigned long pivot;
	enum maple_type type;
	struct maple_node *node;
	unsigned long save_point = mas->last;
	void *entry;

retry:
	node = mas_mn(mas);
	type = mte_node_type(mas->node);
	pivots = ma_pivots(node, type);
	if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
		goto retry;

	if (mas->max >= max) {
		if (likely(mas->offset < mas->end))
			pivot = pivots[mas->offset];
		else
			pivot = mas->max;

		if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
			goto retry;

		if (pivot >= max) { /* Was at the limit, next will extend beyond */
			mas->status = ma_overflow;
			return NULL;
		}
	}

	if (likely(mas->offset < mas->end)) {
		mas->index = pivots[mas->offset] + 1;
again:
		mas->offset++;
		if (likely(mas->offset < mas->end))
			mas->last = pivots[mas->offset];
		else
			mas->last = mas->max;
	} else  {
		if (mas->last >= max) {
			mas->status = ma_overflow;
			return NULL;
		}

		if (mas_next_node(mas, node, max)) {
			mas_rewalk(mas, save_point);
			goto retry;
		}

		if (WARN_ON_ONCE(mas_is_overflow(mas)))
			return NULL;

		mas->offset = 0;
		mas->index = mas->min;
		node = mas_mn(mas);
		type = mte_node_type(mas->node);
		pivots = ma_pivots(node, type);
		mas->last = pivots[0];
	}

	slots = ma_slots(node, type);
	entry = mt_slot(mas->tree, slots, mas->offset);
	if (unlikely(mas_rewalk_if_dead(mas, node, save_point)))
		goto retry;

	if (entry)
		return entry;


	if (!empty) {
		if (mas->last >= max) {
			mas->status = ma_overflow;
			return NULL;
		}

		mas->index = mas->last + 1;
		goto again;
	}

	return entry;
}

/*
 * mas_rev_awalk() - Internal function.  Reverse allocation walk.  Find the
 * highest gap address of a given size in a given node and descend.
 * @mas: The maple state
 * @size: The needed size.
 *
 * Return: True if found in a leaf, false otherwise.
 *
 */
static bool mas_rev_awalk(struct ma_state *mas, unsigned long size,
		unsigned long *gap_min, unsigned long *gap_max)
{
	enum maple_type type = mte_node_type(mas->node);
	struct maple_node *node = mas_mn(mas);
	unsigned long *pivots, *gaps;
	void __rcu **slots;
	unsigned long gap = 0;
	unsigned long max, min;
	unsigned char offset;

	if (unlikely(mas_is_err(mas)))
		return true;

	if (ma_is_dense(type)) {
		/* dense nodes. */
		mas->offset = (unsigned char)(mas->index - mas->min);
		return true;
	}

	pivots = ma_pivots(node, type);
	slots = ma_slots(node, type);
	gaps = ma_gaps(node, type);
	offset = mas->offset;
	min = mas_safe_min(mas, pivots, offset);
	/* Skip out of bounds. */
	while (mas->last < min)
		min = mas_safe_min(mas, pivots, --offset);

	max = mas_safe_pivot(mas, pivots, offset, type);
	while (mas->index <= max) {
		gap = 0;
		if (gaps)
			gap = gaps[offset];
		else if (!mas_slot(mas, slots, offset))
			gap = max - min + 1;

		if (gap) {
			if ((size <= gap) && (size <= mas->last - min + 1))
				break;

			if (!gaps) {
				/* Skip the next slot, it cannot be a gap. */
				if (offset < 2)
					goto ascend;

				offset -= 2;
				max = pivots[offset];
				min = mas_safe_min(mas, pivots, offset);
				continue;
			}
		}

		if (!offset)
			goto ascend;

		offset--;
		max = min - 1;
		min = mas_safe_min(mas, pivots, offset);
	}

	if (unlikely((mas->index > max) || (size - 1 > max - mas->index)))
		goto no_space;

	if (unlikely(ma_is_leaf(type))) {
		mas->offset = offset;
		*gap_min = min;
		*gap_max = min + gap - 1;
		return true;
	}

	/* descend, only happens under lock. */
	mas->node = mas_slot(mas, slots, offset);
	mas->min = min;
	mas->max = max;
	mas->offset = mas_data_end(mas);
	return false;

ascend:
	if (!mte_is_root(mas->node))
		return false;

no_space:
	mas_set_err(mas, -EBUSY);
	return false;
}

static inline bool mas_anode_descend(struct ma_state *mas, unsigned long size)
{
	enum maple_type type = mte_node_type(mas->node);
	unsigned long pivot, min, gap = 0;
	unsigned char offset, data_end;
	unsigned long *gaps, *pivots;
	void __rcu **slots;
	struct maple_node *node;
	bool found = false;

	if (ma_is_dense(type)) {
		mas->offset = (unsigned char)(mas->index - mas->min);
		return true;
	}

	node = mas_mn(mas);
	pivots = ma_pivots(node, type);
	slots = ma_slots(node, type);
	gaps = ma_gaps(node, type);
	offset = mas->offset;
	min = mas_safe_min(mas, pivots, offset);
	data_end = ma_data_end(node, type, pivots, mas->max);
	for (; offset <= data_end; offset++) {
		pivot = mas_safe_pivot(mas, pivots, offset, type);

		/* Not within lower bounds */
		if (mas->index > pivot)
			goto next_slot;

		if (gaps)
			gap = gaps[offset];
		else if (!mas_slot(mas, slots, offset))
			gap = min(pivot, mas->last) - max(mas->index, min) + 1;
		else
			goto next_slot;

		if (gap >= size) {
			if (ma_is_leaf(type)) {
				found = true;
				break;
			}

			mas->node = mas_slot(mas, slots, offset);
			mas->min = min;
			mas->max = pivot;
			offset = 0;
			break;
		}
next_slot:
		min = pivot + 1;
		if (mas->last <= pivot) {
			mas_set_err(mas, -EBUSY);
			return true;
		}
	}

	mas->offset = offset;
	return found;
}

/**
 * mas_walk() - Search for @mas->index in the tree.
 * @mas: The maple state.
 *
 * mas->index and mas->last will be set to the range if there is a value.  If
 * mas->status is ma_none, reset to ma_start
 *
 * Return: the entry at the location or %NULL.
 */
void *mas_walk(struct ma_state *mas)
{
	void *entry;

	if (!mas_is_active(mas) && !mas_is_start(mas))
		mas->status = ma_start;
retry:
	entry = mas_state_walk(mas);
	if (mas_is_start(mas)) {
		goto retry;
	} else if (mas_is_none(mas)) {
		mas->index = 0;
		mas->last = ULONG_MAX;
	} else if (mas_is_ptr(mas)) {
		if (!mas->index) {
			mas->last = 0;
			return entry;
		}

		mas->index = 1;
		mas->last = ULONG_MAX;
		mas->status = ma_none;
		return NULL;
	}

	return entry;
}
EXPORT_SYMBOL_GPL(mas_walk);

static inline bool mas_rewind_node(struct ma_state *mas)
{
	unsigned char slot;

	do {
		if (mte_is_root(mas->node)) {
			slot = mas->offset;
			if (!slot)
				return false;
		} else {
			mas_ascend(mas);
			slot = mas->offset;
		}
	} while (!slot);

	mas->offset = --slot;
	return true;
}

/*
 * mas_skip_node() - Internal function.  Skip over a node.
 * @mas: The maple state.
 *
 * Return: true if there is another node, false otherwise.
 */
static inline bool mas_skip_node(struct ma_state *mas)
{
	if (mas_is_err(mas))
		return false;

	do {
		if (mte_is_root(mas->node)) {
			if (mas->offset >= mas_data_end(mas)) {
				mas_set_err(mas, -EBUSY);
				return false;
			}
		} else {
			mas_ascend(mas);
		}
	} while (mas->offset >= mas_data_end(mas));

	mas->offset++;
	return true;
}

/*
 * mas_awalk() - Allocation walk.  Search from low address to high, for a gap of
 * @size
 * @mas: The maple state
 * @size: The size of the gap required
 *
 * Search between @mas->index and @mas->last for a gap of @size.
 */
static inline void mas_awalk(struct ma_state *mas, unsigned long size)
{
	struct maple_enode *last = NULL;

	/*
	 * There are 4 options:
	 * go to child (descend)
	 * go back to parent (ascend)
	 * no gap found. (return, error == -EBUSY)
	 * found the gap. (return)
	 */
	while (!mas_is_err(mas) && !mas_anode_descend(mas, size)) {
		if (last == mas->node)
			mas_skip_node(mas);
		else
			last = mas->node;
	}
}

/*
 * mas_sparse_area() - Internal function.  Return upper or lower limit when
 * searching for a gap in an empty tree.
 * @mas: The maple state
 * @min: the minimum range
 * @max: The maximum range
 * @size: The size of the gap
 * @fwd: Searching forward or back
 */
static inline int mas_sparse_area(struct ma_state *mas, unsigned long min,
				unsigned long max, unsigned long size, bool fwd)
{
	if (!unlikely(mas_is_none(mas)) && min == 0) {
		min++;
		/*
		 * At this time, min is increased, we need to recheck whether
		 * the size is satisfied.
		 */
		if (min > max || max - min + 1 < size)
			return -EBUSY;
	}
	/* mas_is_ptr */

	if (fwd) {
		mas->index = min;
		mas->last = min + size - 1;
	} else {
		mas->last = max;
		mas->index = max - size + 1;
	}
	return 0;
}

/*
 * mas_empty_area() - Get the lowest address within the range that is
 * sufficient for the size requested.
 * @mas: The maple state
 * @min: The lowest value of the range
 * @max: The highest value of the range
 * @size: The size needed
 */
int mas_empty_area(struct ma_state *mas, unsigned long min,
		unsigned long max, unsigned long size)
{
	unsigned char offset;
	unsigned long *pivots;
	enum maple_type mt;
	struct maple_node *node;

	if (min > max)
		return -EINVAL;

	if (size == 0 || max - min < size - 1)
		return -EINVAL;

	if (mas_is_start(mas))
		mas_start(mas);
	else if (mas->offset >= 2)
		mas->offset -= 2;
	else if (!mas_skip_node(mas))
		return -EBUSY;

	/* Empty set */
	if (mas_is_none(mas) || mas_is_ptr(mas))
		return mas_sparse_area(mas, min, max, size, true);

	/* The start of the window can only be within these values */
	mas->index = min;
	mas->last = max;
	mas_awalk(mas, size);

	if (unlikely(mas_is_err(mas)))
		return xa_err(mas->node);

	offset = mas->offset;
	node = mas_mn(mas);
	mt = mte_node_type(mas->node);
	pivots = ma_pivots(node, mt);
	min = mas_safe_min(mas, pivots, offset);
	if (mas->index < min)
		mas->index = min;
	mas->last = mas->index + size - 1;
	mas->end = ma_data_end(node, mt, pivots, mas->max);
	return 0;
}
EXPORT_SYMBOL_GPL(mas_empty_area);

/*
 * mas_empty_area_rev() - Get the highest address within the range that is
 * sufficient for the size requested.
 * @mas: The maple state
 * @min: The lowest value of the range
 * @max: The highest value of the range
 * @size: The size needed
 */
int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
		unsigned long max, unsigned long size)
{
	struct maple_enode *last = mas->node;

	if (min > max)
		return -EINVAL;

	if (size == 0 || max - min < size - 1)
		return -EINVAL;

	if (mas_is_start(mas))
		mas_start(mas);
	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
		return -EBUSY;

	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
		return mas_sparse_area(mas, min, max, size, false);
	else if (mas->offset >= 2)
		mas->offset -= 2;
	else
		mas->offset = mas_data_end(mas);


	/* The start of the window can only be within these values. */
	mas->index = min;
	mas->last = max;

	while (!mas_rev_awalk(mas, size, &min, &max)) {
		if (last == mas->node) {
			if (!mas_rewind_node(mas))
				return -EBUSY;
		} else {
			last = mas->node;
		}
	}

	if (mas_is_err(mas))
		return xa_err(mas->node);

	if (unlikely(mas->offset == MAPLE_NODE_SLOTS))
		return -EBUSY;

	/* Trim the upper limit to the max. */
	if (max < mas->last)
		mas->last = max;

	mas->index = mas->last - size + 1;
	mas->end = mas_data_end(mas);
	return 0;
}
EXPORT_SYMBOL_GPL(mas_empty_area_rev);

/*
 * mte_dead_leaves() - Mark all leaves of a node as dead.
 * @enode: the encoded node
 * @mt: the maple tree
 * @slots: Pointer to the slot array
 *
 * Must hold the write lock.
 *
 * Return: The number of leaves marked as dead.
 */
static inline
unsigned char mte_dead_leaves(struct maple_enode *enode, struct maple_tree *mt,
			      void __rcu **slots)
{
	struct maple_node *node;
	enum maple_type type;
	void *entry;
	int offset;

	for (offset = 0; offset < mt_slot_count(enode); offset++) {
		entry = mt_slot(mt, slots, offset);
		type = mte_node_type(entry);
		node = mte_to_node(entry);
		/* Use both node and type to catch LE & BE metadata */
		if (!node || !type)
			break;

		mte_set_node_dead(entry);
		node->type = type;
		rcu_assign_pointer(slots[offset], node);
	}

	return offset;
}

/**
 * mte_dead_walk() - Walk down a dead tree to just before the leaves
 * @enode: The maple encoded node
 * @offset: The starting offset
 *
 * Note: This can only be used from the RCU callback context.
 */
static void __rcu **mte_dead_walk(struct maple_enode **enode, unsigned char offset)
{
	struct maple_node *node, *next;
	void __rcu **slots = NULL;

	next = mte_to_node(*enode);
	do {
		*enode = ma_enode_ptr(next);
		node = mte_to_node(*enode);
		slots = ma_slots(node, node->type);
		next = rcu_dereference_protected(slots[offset],
					lock_is_held(&rcu_callback_map));
		offset = 0;
	} while (!ma_is_leaf(next->type));

	return slots;
}

/**
 * mt_free_walk() - Walk & free a tree in the RCU callback context
 * @head: The RCU head that's within the node.
 *
 * Note: This can only be used from the RCU callback context.
 */
static void mt_free_walk(struct rcu_head *head)
{
	void __rcu **slots;
	struct maple_node *node, *start;
	struct maple_enode *enode;
	unsigned char offset;
	enum maple_type type;

	node = container_of(head, struct maple_node, rcu);

	if (ma_is_leaf(node->type))
		goto free_leaf;

	start = node;
	enode = mt_mk_node(node, node->type);
	slots = mte_dead_walk(&enode, 0);
	node = mte_to_node(enode);
	do {
		mt_free_bulk(node->slot_len, slots);
		offset = node->parent_slot + 1;
		enode = node->piv_parent;
		if (mte_to_node(enode) == node)
			goto free_leaf;

		type = mte_node_type(enode);
		slots = ma_slots(mte_to_node(enode), type);
		if ((offset < mt_slots[type]) &&
		    rcu_dereference_protected(slots[offset],
					      lock_is_held(&rcu_callback_map)))
			slots = mte_dead_walk(&enode, offset);
		node = mte_to_node(enode);
	} while ((node != start) || (node->slot_len < offset));

	slots = ma_slots(node, node->type);
	mt_free_bulk(node->slot_len, slots);

free_leaf:
	kfree(node);
}

static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
	struct maple_tree *mt, struct maple_enode *prev, unsigned char offset)
{
	struct maple_node *node;
	struct maple_enode *next = *enode;
	void __rcu **slots = NULL;
	enum maple_type type;
	unsigned char next_offset = 0;

	do {
		*enode = next;
		node = mte_to_node(*enode);
		type = mte_node_type(*enode);
		slots = ma_slots(node, type);
		next = mt_slot_locked(mt, slots, next_offset);
		if ((mte_dead_node(next)))
			next = mt_slot_locked(mt, slots, ++next_offset);

		mte_set_node_dead(*enode);
		node->type = type;
		node->piv_parent = prev;
		node->parent_slot = offset;
		offset = next_offset;
		next_offset = 0;
		prev = *enode;
	} while (!mte_is_leaf(next));

	return slots;
}

static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
			    bool free)
{
	void __rcu **slots;
	struct maple_node *node = mte_to_node(enode);
	struct maple_enode *start;

	if (mte_is_leaf(enode)) {
		mte_set_node_dead(enode);
		node->type = mte_node_type(enode);
		goto free_leaf;
	}

	start = enode;
	slots = mte_destroy_descend(&enode, mt, start, 0);
	node = mte_to_node(enode); // Updated in the above call.
	do {
		enum maple_type type;
		unsigned char offset;
		struct maple_enode *parent, *tmp;

		node->slot_len = mte_dead_leaves(enode, mt, slots);
		if (free)
			mt_free_bulk(node->slot_len, slots);
		offset = node->parent_slot + 1;
		enode = node->piv_parent;
		if (mte_to_node(enode) == node)
			goto free_leaf;

		type = mte_node_type(enode);
		slots = ma_slots(mte_to_node(enode), type);
		if (offset >= mt_slots[type])
			goto next;

		tmp = mt_slot_locked(mt, slots, offset);
		if (mte_node_type(tmp) && mte_to_node(tmp)) {
			parent = enode;
			enode = tmp;
			slots = mte_destroy_descend(&enode, mt, parent, offset);
		}
next:
		node = mte_to_node(enode);
	} while (start != enode);

	node = mte_to_node(enode);
	node->slot_len = mte_dead_leaves(enode, mt, slots);
	if (free)
		mt_free_bulk(node->slot_len, slots);

free_leaf:
	if (free)
		kfree(node);
	else
		mt_clear_meta(mt, node, node->type);
}

/*
 * mte_destroy_walk() - Free a tree or sub-tree.
 * @enode: the encoded maple node (maple_enode) to start
 * @mt: the tree to free - needed for node types.
 *
 * Must hold the write lock.
 */
static inline void mte_destroy_walk(struct maple_enode *enode,
				    struct maple_tree *mt)
{
	struct maple_node *node = mte_to_node(enode);

	if (mt_in_rcu(mt)) {
		mt_destroy_walk(enode, mt, false);
		call_rcu(&node->rcu, mt_free_walk);
	} else {
		mt_destroy_walk(enode, mt, true);
	}
}
/* Interface */

/**
 * mas_store() - Store an @entry.
 * @mas: The maple state.
 * @entry: The entry to store.
 *
 * The @mas->index and @mas->last is used to set the range for the @entry.
 *
 * Return: the first entry between mas->index and mas->last or %NULL.
 */
void *mas_store(struct ma_state *mas, void *entry)
{
	MA_WR_STATE(wr_mas, mas, entry);

	trace_ma_write(TP_FCT, mas, 0, entry);
#ifdef CONFIG_DEBUG_MAPLE_TREE
	if (MAS_WARN_ON(mas, mas->index > mas->last))
		pr_err("Error %lX > %lX " PTR_FMT "\n", mas->index, mas->last,
		       entry);

	if (mas->index > mas->last) {
		mas_set_err(mas, -EINVAL);
		return NULL;
	}

#endif

	/*
	 * Storing is the same operation as insert with the added caveat that it
	 * can overwrite entries.  Although this seems simple enough, one may
	 * want to examine what happens if a single store operation was to
	 * overwrite multiple entries within a self-balancing B-Tree.
	 */
	mas_wr_prealloc_setup(&wr_mas);
	mas->store_type = mas_wr_store_type(&wr_mas);
	if (mas->mas_flags & MA_STATE_PREALLOC) {
		mas_wr_store_entry(&wr_mas);
		MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas));
		return wr_mas.content;
	}

	mas_prealloc_calc(&wr_mas, entry);
	if (!mas->node_request)
		goto store;

	mas_alloc_nodes(mas, GFP_NOWAIT);
	if (mas_is_err(mas))
		return NULL;

store:
	mas_wr_store_entry(&wr_mas);
	mas_destroy(mas);
	return wr_mas.content;
}
EXPORT_SYMBOL_GPL(mas_store);

/**
 * mas_store_gfp() - Store a value into the tree.
 * @mas: The maple state
 * @entry: The entry to store
 * @gfp: The GFP_FLAGS to use for allocations if necessary.
 *
 * Return: 0 on success, -EINVAL on invalid request, -ENOMEM if memory could not
 * be allocated.
 */
int mas_store_gfp(struct ma_state *mas, void *entry, gfp_t gfp)
{
	unsigned long index = mas->index;
	unsigned long last = mas->last;
	MA_WR_STATE(wr_mas, mas, entry);
	int ret = 0;

retry:
	mas_wr_preallocate(&wr_mas, entry);
	if (unlikely(mas_nomem(mas, gfp))) {
		if (!entry)
			__mas_set_range(mas, index, last);
		goto retry;
	}

	if (mas_is_err(mas)) {
		ret = xa_err(mas->node);
		goto out;
	}

	mas_wr_store_entry(&wr_mas);
out:
	mas_destroy(mas);
	return ret;
}
EXPORT_SYMBOL_GPL(mas_store_gfp);

/**
 * mas_store_prealloc() - Store a value into the tree using memory
 * preallocated in the maple state.
 * @mas: The maple state
 * @entry: The entry to store.
 */
void mas_store_prealloc(struct ma_state *mas, void *entry)
{
	MA_WR_STATE(wr_mas, mas, entry);

	if (mas->store_type == wr_store_root) {
		mas_wr_prealloc_setup(&wr_mas);
		goto store;
	}

	mas_wr_walk_descend(&wr_mas);
	if (mas->store_type != wr_spanning_store) {
		/* set wr_mas->content to current slot */
		wr_mas.content = mas_slot_locked(mas, wr_mas.slots, mas->offset);
		mas_wr_end_piv(&wr_mas);
	}

store:
	trace_ma_write(TP_FCT, mas, 0, entry);
	mas_wr_store_entry(&wr_mas);
	MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas));
	mas_destroy(mas);
}
EXPORT_SYMBOL_GPL(mas_store_prealloc);

/**
 * mas_preallocate() - Preallocate enough nodes for a store operation
 * @mas: The maple state
 * @entry: The entry that will be stored
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * Return: 0 on success, -ENOMEM if memory could not be allocated.
 */
int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
{
	MA_WR_STATE(wr_mas, mas, entry);

	mas_wr_prealloc_setup(&wr_mas);
	mas->store_type = mas_wr_store_type(&wr_mas);
	mas_prealloc_calc(&wr_mas, entry);
	if (!mas->node_request)
		goto set_flag;

	mas->mas_flags &= ~MA_STATE_PREALLOC;
	mas_alloc_nodes(mas, gfp);
	if (mas_is_err(mas)) {
		int ret = xa_err(mas->node);

		mas->node_request = 0;
		mas_destroy(mas);
		mas_reset(mas);
		return ret;
	}

set_flag:
	mas->mas_flags |= MA_STATE_PREALLOC;
	return 0;
}
EXPORT_SYMBOL_GPL(mas_preallocate);

/*
 * mas_destroy() - destroy a maple state.
 * @mas: The maple state
 *
 * Upon completion, check the left-most node and rebalance against the node to
 * the right if necessary.  Frees any allocated nodes associated with this maple
 * state.
 */
void mas_destroy(struct ma_state *mas)
{
	mas->mas_flags &= ~MA_STATE_PREALLOC;
	mas_empty_nodes(mas);
}
EXPORT_SYMBOL_GPL(mas_destroy);

static void mas_may_activate(struct ma_state *mas)
{
	if (!mas->node) {
		mas->status = ma_start;
	} else if (mas->index > mas->max || mas->index < mas->min) {
		mas->status = ma_start;
	} else {
		mas->status = ma_active;
	}
}

static bool mas_next_setup(struct ma_state *mas, unsigned long max,
		void **entry)
{
	bool was_none = mas_is_none(mas);

	if (unlikely(mas->last >= max)) {
		mas->status = ma_overflow;
		return true;
	}

	switch (mas->status) {
	case ma_active:
		return false;
	case ma_none:
		fallthrough;
	case ma_pause:
		mas->status = ma_start;
		fallthrough;
	case ma_start:
		mas_walk(mas); /* Retries on dead nodes handled by mas_walk */
		break;
	case ma_overflow:
		/* Overflowed before, but the max changed */
		mas_may_activate(mas);
		break;
	case ma_underflow:
		/* The user expects the mas to be one before where it is */
		mas_may_activate(mas);
		*entry = mas_walk(mas);
		if (*entry)
			return true;
		break;
	case ma_root:
		break;
	case ma_error:
		return true;
	}

	if (likely(mas_is_active(mas))) /* Fast path */
		return false;

	if (mas_is_ptr(mas)) {
		*entry = NULL;
		if (was_none && mas->index == 0) {
			mas->index = mas->last = 0;
			return true;
		}
		mas->index = 1;
		mas->last = ULONG_MAX;
		mas->status = ma_none;
		return true;
	}

	if (mas_is_none(mas))
		return true;

	return false;
}

/**
 * mas_next() - Get the next entry.
 * @mas: The maple state
 * @max: The maximum index to check.
 *
 * Returns the next entry after @mas->index.
 * Must hold rcu_read_lock or the write lock.
 * Can return the zero entry.
 *
 * Return: The next entry or %NULL
 */
void *mas_next(struct ma_state *mas, unsigned long max)
{
	void *entry = NULL;

	if (mas_next_setup(mas, max, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_next_slot */
	return mas_next_slot(mas, max, false);
}
EXPORT_SYMBOL_GPL(mas_next);

/**
 * mas_next_range() - Advance the maple state to the next range
 * @mas: The maple state
 * @max: The maximum index to check.
 *
 * Sets @mas->index and @mas->last to the range.
 * Must hold rcu_read_lock or the write lock.
 * Can return the zero entry.
 *
 * Return: The next entry or %NULL
 */
void *mas_next_range(struct ma_state *mas, unsigned long max)
{
	void *entry = NULL;

	if (mas_next_setup(mas, max, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_next_slot */
	return mas_next_slot(mas, max, true);
}
EXPORT_SYMBOL_GPL(mas_next_range);

/**
 * mt_next() - get the next value in the maple tree
 * @mt: The maple tree
 * @index: The start index
 * @max: The maximum index to check
 *
 * Takes RCU read lock internally to protect the search, which does not
 * protect the returned pointer after dropping RCU read lock.
 * See also: Documentation/core-api/maple_tree.rst
 *
 * Return: The entry higher than @index or %NULL if nothing is found.
 */
void *mt_next(struct maple_tree *mt, unsigned long index, unsigned long max)
{
	void *entry = NULL;
	MA_STATE(mas, mt, index, index);

	rcu_read_lock();
	entry = mas_next(&mas, max);
	rcu_read_unlock();
	return entry;
}
EXPORT_SYMBOL_GPL(mt_next);

static bool mas_prev_setup(struct ma_state *mas, unsigned long min, void **entry)
{
	if (unlikely(mas->index <= min)) {
		mas->status = ma_underflow;
		return true;
	}

	switch (mas->status) {
	case ma_active:
		return false;
	case ma_start:
		break;
	case ma_none:
		fallthrough;
	case ma_pause:
		mas->status = ma_start;
		break;
	case ma_underflow:
		/* underflowed before but the min changed */
		mas_may_activate(mas);
		break;
	case ma_overflow:
		/* User expects mas to be one after where it is */
		mas_may_activate(mas);
		*entry = mas_walk(mas);
		if (*entry)
			return true;
		break;
	case ma_root:
		break;
	case ma_error:
		return true;
	}

	if (mas_is_start(mas))
		mas_walk(mas);

	if (unlikely(mas_is_ptr(mas))) {
		if (!mas->index) {
			mas->status = ma_none;
			return true;
		}
		mas->index = mas->last = 0;
		*entry = mas_root(mas);
		return true;
	}

	if (mas_is_none(mas)) {
		if (mas->index) {
			/* Walked to out-of-range pointer? */
			mas->index = mas->last = 0;
			mas->status = ma_root;
			*entry = mas_root(mas);
			return true;
		}
		return true;
	}

	return false;
}

/**
 * mas_prev() - Get the previous entry
 * @mas: The maple state
 * @min: The minimum value to check.
 *
 * Must hold rcu_read_lock or the write lock.
 * Will reset mas to ma_start if the status is ma_none.  Will stop on not
 * searchable nodes.
 *
 * Return: the previous value or %NULL.
 */
void *mas_prev(struct ma_state *mas, unsigned long min)
{
	void *entry = NULL;

	if (mas_prev_setup(mas, min, &entry))
		return entry;

	return mas_prev_slot(mas, min, false);
}
EXPORT_SYMBOL_GPL(mas_prev);

/**
 * mas_prev_range() - Advance to the previous range
 * @mas: The maple state
 * @min: The minimum value to check.
 *
 * Sets @mas->index and @mas->last to the range.
 * Must hold rcu_read_lock or the write lock.
 * Will reset mas to ma_start if the node is ma_none.  Will stop on not
 * searchable nodes.
 *
 * Return: the previous value or %NULL.
 */
void *mas_prev_range(struct ma_state *mas, unsigned long min)
{
	void *entry = NULL;

	if (mas_prev_setup(mas, min, &entry))
		return entry;

	return mas_prev_slot(mas, min, true);
}
EXPORT_SYMBOL_GPL(mas_prev_range);

/**
 * mt_prev() - get the previous value in the maple tree
 * @mt: The maple tree
 * @index: The start index
 * @min: The minimum index to check
 *
 * Takes RCU read lock internally to protect the search, which does not
 * protect the returned pointer after dropping RCU read lock.
 * See also: Documentation/core-api/maple_tree.rst
 *
 * Return: The entry before @index or %NULL if nothing is found.
 */
void *mt_prev(struct maple_tree *mt, unsigned long index, unsigned long min)
{
	void *entry = NULL;
	MA_STATE(mas, mt, index, index);

	rcu_read_lock();
	entry = mas_prev(&mas, min);
	rcu_read_unlock();
	return entry;
}
EXPORT_SYMBOL_GPL(mt_prev);

/**
 * mas_pause() - Pause a mas_find/mas_for_each to drop the lock.
 * @mas: The maple state to pause
 *
 * Some users need to pause a walk and drop the lock they're holding in
 * order to yield to a higher priority thread or carry out an operation
 * on an entry.  Those users should call this function before they drop
 * the lock.  It resets the @mas to be suitable for the next iteration
 * of the loop after the user has reacquired the lock.  If most entries
 * found during a walk require you to call mas_pause(), the mt_for_each()
 * iterator may be more appropriate.
 *
 */
void mas_pause(struct ma_state *mas)
{
	mas->status = ma_pause;
	mas->node = NULL;
}
EXPORT_SYMBOL_GPL(mas_pause);

/**
 * mas_find_setup() - Internal function to set up mas_find*().
 * @mas: The maple state
 * @max: The maximum index
 * @entry: Pointer to the entry
 *
 * Returns: True if entry is the answer, false otherwise.
 */
static __always_inline bool mas_find_setup(struct ma_state *mas, unsigned long max, void **entry)
{
	switch (mas->status) {
	case ma_active:
		if (mas->last < max)
			return false;
		return true;
	case ma_start:
		break;
	case ma_pause:
		if (unlikely(mas->last >= max))
			return true;

		mas->index = ++mas->last;
		mas->status = ma_start;
		break;
	case ma_none:
		if (unlikely(mas->last >= max))
			return true;

		mas->index = mas->last;
		mas->status = ma_start;
		break;
	case ma_underflow:
		/* mas is pointing at entry before unable to go lower */
		if (unlikely(mas->index >= max)) {
			mas->status = ma_overflow;
			return true;
		}

		mas_may_activate(mas);
		*entry = mas_walk(mas);
		if (*entry)
			return true;
		break;
	case ma_overflow:
		if (unlikely(mas->last >= max))
			return true;

		mas_may_activate(mas);
		*entry = mas_walk(mas);
		if (*entry)
			return true;
		break;
	case ma_root:
		break;
	case ma_error:
		return true;
	}

	if (mas_is_start(mas)) {
		/* First run or continue */
		if (mas->index > max)
			return true;

		*entry = mas_walk(mas);
		if (*entry)
			return true;

	}

	if (unlikely(mas_is_ptr(mas)))
		goto ptr_out_of_range;

	if (unlikely(mas_is_none(mas)))
		return true;

	if (mas->index == max)
		return true;

	return false;

ptr_out_of_range:
	mas->status = ma_none;
	mas->index = 1;
	mas->last = ULONG_MAX;
	return true;
}

/**
 * mas_find() - On the first call, find the entry at or after mas->index up to
 * %max.  Otherwise, find the entry after mas->index.
 * @mas: The maple state
 * @max: The maximum value to check.
 *
 * Must hold rcu_read_lock or the write lock.
 * If an entry exists, last and index are updated accordingly.
 * May set @mas->status to ma_overflow.
 *
 * Return: The entry or %NULL.
 */
void *mas_find(struct ma_state *mas, unsigned long max)
{
	void *entry = NULL;

	if (mas_find_setup(mas, max, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_next_slot */
	entry = mas_next_slot(mas, max, false);
	/* Ignore overflow */
	mas->status = ma_active;
	return entry;
}
EXPORT_SYMBOL_GPL(mas_find);

/**
 * mas_find_range() - On the first call, find the entry at or after
 * mas->index up to %max.  Otherwise, advance to the next slot mas->index.
 * @mas: The maple state
 * @max: The maximum value to check.
 *
 * Must hold rcu_read_lock or the write lock.
 * If an entry exists, last and index are updated accordingly.
 * May set @mas->status to ma_overflow.
 *
 * Return: The entry or %NULL.
 */
void *mas_find_range(struct ma_state *mas, unsigned long max)
{
	void *entry = NULL;

	if (mas_find_setup(mas, max, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_next_slot */
	return mas_next_slot(mas, max, true);
}
EXPORT_SYMBOL_GPL(mas_find_range);

/**
 * mas_find_rev_setup() - Internal function to set up mas_find_*_rev()
 * @mas: The maple state
 * @min: The minimum index
 * @entry: Pointer to the entry
 *
 * Returns: True if entry is the answer, false otherwise.
 */
static bool mas_find_rev_setup(struct ma_state *mas, unsigned long min,
		void **entry)
{

	switch (mas->status) {
	case ma_active:
		goto active;
	case ma_start:
		break;
	case ma_pause:
		if (unlikely(mas->index <= min)) {
			mas->status = ma_underflow;
			return true;
		}
		mas->last = --mas->index;
		mas->status = ma_start;
		break;
	case ma_none:
		if (mas->index <= min)
			goto none;

		mas->last = mas->index;
		mas->status = ma_start;
		break;
	case ma_overflow: /* user expects the mas to be one after where it is */
		if (unlikely(mas->index <= min)) {
			mas->status = ma_underflow;
			return true;
		}

		mas->status = ma_active;
		break;
	case ma_underflow: /* user expects the mas to be one before where it is */
		if (unlikely(mas->index <= min))
			return true;

		mas->status = ma_active;
		break;
	case ma_root:
		break;
	case ma_error:
		return true;
	}

	if (mas_is_start(mas)) {
		/* First run or continue */
		if (mas->index < min)
			return true;

		*entry = mas_walk(mas);
		if (*entry)
			return true;
	}

	if (unlikely(mas_is_ptr(mas)))
		goto none;

	if (unlikely(mas_is_none(mas))) {
		/*
		 * Walked to the location, and there was nothing so the previous
		 * location is 0.
		 */
		mas->last = mas->index = 0;
		mas->status = ma_root;
		*entry = mas_root(mas);
		return true;
	}

active:
	if (mas->index < min)
		return true;

	return false;

none:
	mas->status = ma_none;
	return true;
}

/**
 * mas_find_rev: On the first call, find the first non-null entry at or below
 * mas->index down to %min.  Otherwise find the first non-null entry below
 * mas->index down to %min.
 * @mas: The maple state
 * @min: The minimum value to check.
 *
 * Must hold rcu_read_lock or the write lock.
 * If an entry exists, last and index are updated accordingly.
 * May set @mas->status to ma_underflow.
 *
 * Return: The entry or %NULL.
 */
void *mas_find_rev(struct ma_state *mas, unsigned long min)
{
	void *entry = NULL;

	if (mas_find_rev_setup(mas, min, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_prev_slot */
	return mas_prev_slot(mas, min, false);

}
EXPORT_SYMBOL_GPL(mas_find_rev);

/**
 * mas_find_range_rev: On the first call, find the first non-null entry at or
 * below mas->index down to %min.  Otherwise advance to the previous slot after
 * mas->index down to %min.
 * @mas: The maple state
 * @min: The minimum value to check.
 *
 * Must hold rcu_read_lock or the write lock.
 * If an entry exists, last and index are updated accordingly.
 * May set @mas->status to ma_underflow.
 *
 * Return: The entry or %NULL.
 */
void *mas_find_range_rev(struct ma_state *mas, unsigned long min)
{
	void *entry = NULL;

	if (mas_find_rev_setup(mas, min, &entry))
		return entry;

	/* Retries on dead nodes handled by mas_prev_slot */
	return mas_prev_slot(mas, min, true);
}
EXPORT_SYMBOL_GPL(mas_find_range_rev);

/**
 * mas_erase() - Find the range in which index resides and erase the entire
 * range.
 * @mas: The maple state
 *
 * Must hold the write lock.
 * Searches for @mas->index, sets @mas->index and @mas->last to the range and
 * erases that range.
 *
 * Return: the entry that was erased or %NULL, @mas->index and @mas->last are updated.
 */
void *mas_erase(struct ma_state *mas)
{
	void *entry;
	unsigned long index = mas->index;
	MA_WR_STATE(wr_mas, mas, NULL);

	if (!mas_is_active(mas) || !mas_is_start(mas))
		mas->status = ma_start;

write_retry:
	entry = mas_state_walk(mas);
	if (!entry)
		return NULL;

	/* Must reset to ensure spanning writes of last slot are detected */
	mas_reset(mas);
	mas_wr_preallocate(&wr_mas, NULL);
	if (mas_nomem(mas, GFP_KERNEL)) {
		/* in case the range of entry changed when unlocked */
		mas->index = mas->last = index;
		goto write_retry;
	}

	if (mas_is_err(mas))
		goto out;

	mas_wr_store_entry(&wr_mas);
out:
	mas_destroy(mas);
	return entry;
}
EXPORT_SYMBOL_GPL(mas_erase);

/**
 * mas_nomem() - Check if there was an error allocating and do the allocation
 * if necessary If there are allocations, then free them.
 * @mas: The maple state
 * @gfp: The GFP_FLAGS to use for allocations
 * Return: true on allocation, false otherwise.
 */
bool mas_nomem(struct ma_state *mas, gfp_t gfp)
	__must_hold(mas->tree->ma_lock)
{
	if (likely(mas->node != MA_ERROR(-ENOMEM)))
		return false;

	if (gfpflags_allow_blocking(gfp) && !mt_external_lock(mas->tree)) {
		mtree_unlock(mas->tree);
		mas_alloc_nodes(mas, gfp);
		mtree_lock(mas->tree);
	} else {
		mas_alloc_nodes(mas, gfp);
	}

	if (!mas->sheaf && !mas->alloc)
		return false;

	mas->status = ma_start;
	return true;
}

void __init maple_tree_init(void)
{
	struct kmem_cache_args args = {
		.align  = sizeof(struct maple_node),
		.sheaf_capacity = 32,
	};

	maple_node_cache = kmem_cache_create("maple_node",
			sizeof(struct maple_node), &args,
			SLAB_PANIC);
}

/**
 * mtree_load() - Load a value stored in a maple tree
 * @mt: The maple tree
 * @index: The index to load
 *
 * Return: the entry or %NULL
 */
void *mtree_load(struct maple_tree *mt, unsigned long index)
{
	MA_STATE(mas, mt, index, index);
	void *entry;

	trace_ma_read(TP_FCT, &mas);
	rcu_read_lock();
retry:
	entry = mas_start(&mas);
	if (unlikely(mas_is_none(&mas)))
		goto unlock;

	if (unlikely(mas_is_ptr(&mas))) {
		if (index)
			entry = NULL;

		goto unlock;
	}

	entry = mtree_lookup_walk(&mas);
	if (!entry && unlikely(mas_is_start(&mas)))
		goto retry;
unlock:
	rcu_read_unlock();
	if (xa_is_zero(entry))
		return NULL;

	return entry;
}
EXPORT_SYMBOL(mtree_load);

/**
 * mtree_store_range() - Store an entry at a given range.
 * @mt: The maple tree
 * @index: The start of the range
 * @last: The end of the range
 * @entry: The entry to store
 * @gfp: The GFP_FLAGS to use for allocations
 *
 * Return: 0 on success, -EINVAL on invalid request, -ENOMEM if memory could not
 * be allocated.
 */
int mtree_store_range(struct maple_tree *mt, unsigned long index,
		unsigned long last, void *entry, gfp_t gfp)
{
	MA_STATE(mas, mt, index, last);
	int ret = 0;

	trace_ma_write(TP_FCT, &mas, 0, entry);
	if (WARN_ON_ONCE(xa_is_advanced(entry)))
		return -EINVAL;

	if (index > last)
		return -EINVAL;

	mtree_lock(mt);
	ret = mas_store_gfp(&mas, entry, gfp);
	mtree_unlock(mt);

	return ret;
}
EXPORT_SYMBOL(mtree_store_range);

/**
 * mtree_store() - Store an entry at a given index.
 * @mt: The maple tree
 * @index: The index to store the value
 * @entry: The entry to store
 * @gfp: The GFP_FLAGS to use for allocations
 *
 * Return: 0 on success, -EINVAL on invalid request, -ENOMEM if memory could not
 * be allocated.
 */
int mtree_store(struct maple_tree *mt, unsigned long index, void *entry,
		 gfp_t gfp)
{
	return mtree_store_range(mt, index, index, entry, gfp);
}
EXPORT_SYMBOL(mtree_store);

/**
 * mtree_insert_range() - Insert an entry at a given range if there is no value.
 * @mt: The maple tree
 * @first: The start of the range
 * @last: The end of the range
 * @entry: The entry to store
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid
 * request, -ENOMEM if memory could not be allocated.
 */
int mtree_insert_range(struct maple_tree *mt, unsigned long first,
		unsigned long last, void *entry, gfp_t gfp)
{
	MA_STATE(ms, mt, first, last);
	int ret = 0;

	if (WARN_ON_ONCE(xa_is_advanced(entry)))
		return -EINVAL;

	if (first > last)
		return -EINVAL;

	mtree_lock(mt);
retry:
	mas_insert(&ms, entry);
	if (mas_nomem(&ms, gfp))
		goto retry;

	mtree_unlock(mt);
	if (mas_is_err(&ms))
		ret = xa_err(ms.node);

	mas_destroy(&ms);
	return ret;
}
EXPORT_SYMBOL(mtree_insert_range);

/**
 * mtree_insert() - Insert an entry at a given index if there is no value.
 * @mt: The maple tree
 * @index : The index to store the value
 * @entry: The entry to store
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid
 * request, -ENOMEM if memory could not be allocated.
 */
int mtree_insert(struct maple_tree *mt, unsigned long index, void *entry,
		 gfp_t gfp)
{
	return mtree_insert_range(mt, index, index, entry, gfp);
}
EXPORT_SYMBOL(mtree_insert);

int mtree_alloc_range(struct maple_tree *mt, unsigned long *startp,
		void *entry, unsigned long size, unsigned long min,
		unsigned long max, gfp_t gfp)
{
	int ret = 0;

	MA_STATE(mas, mt, 0, 0);
	if (!mt_is_alloc(mt))
		return -EINVAL;

	if (WARN_ON_ONCE(mt_is_reserved(entry)))
		return -EINVAL;

	mtree_lock(mt);
retry:
	ret = mas_empty_area(&mas, min, max, size);
	if (ret)
		goto unlock;

	mas_insert(&mas, entry);
	/*
	 * mas_nomem() may release the lock, causing the allocated area
	 * to be unavailable, so try to allocate a free area again.
	 */
	if (mas_nomem(&mas, gfp))
		goto retry;

	if (mas_is_err(&mas))
		ret = xa_err(mas.node);
	else
		*startp = mas.index;

unlock:
	mtree_unlock(mt);
	mas_destroy(&mas);
	return ret;
}
EXPORT_SYMBOL(mtree_alloc_range);

/**
 * mtree_alloc_cyclic() - Find somewhere to store this entry in the tree.
 * @mt: The maple tree.
 * @startp: Pointer to ID.
 * @range_lo: Lower bound of range to search.
 * @range_hi: Upper bound of range to search.
 * @entry: The entry to store.
 * @next: Pointer to next ID to allocate.
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * Finds an empty entry in @mt after @next, stores the new index into
 * the @id pointer, stores the entry at that index, then updates @next.
 *
 * @mt must be initialized with the MT_FLAGS_ALLOC_RANGE flag.
 *
 * Context: Any context.  Takes and releases the mt.lock.  May sleep if
 * the @gfp flags permit.
 *
 * Return: 0 if the allocation succeeded without wrapping, 1 if the
 * allocation succeeded after wrapping, -ENOMEM if memory could not be
 * allocated, -EINVAL if @mt cannot be used, or -EBUSY if there are no
 * free entries.
 */
int mtree_alloc_cyclic(struct maple_tree *mt, unsigned long *startp,
		void *entry, unsigned long range_lo, unsigned long range_hi,
		unsigned long *next, gfp_t gfp)
{
	int ret;

	MA_STATE(mas, mt, 0, 0);

	if (!mt_is_alloc(mt))
		return -EINVAL;
	if (WARN_ON_ONCE(mt_is_reserved(entry)))
		return -EINVAL;
	mtree_lock(mt);
	ret = mas_alloc_cyclic(&mas, startp, entry, range_lo, range_hi,
			       next, gfp);
	mtree_unlock(mt);
	return ret;
}
EXPORT_SYMBOL(mtree_alloc_cyclic);

int mtree_alloc_rrange(struct maple_tree *mt, unsigned long *startp,
		void *entry, unsigned long size, unsigned long min,
		unsigned long max, gfp_t gfp)
{
	int ret = 0;

	MA_STATE(mas, mt, 0, 0);
	if (!mt_is_alloc(mt))
		return -EINVAL;

	if (WARN_ON_ONCE(mt_is_reserved(entry)))
		return -EINVAL;

	mtree_lock(mt);
retry:
	ret = mas_empty_area_rev(&mas, min, max, size);
	if (ret)
		goto unlock;

	mas_insert(&mas, entry);
	/*
	 * mas_nomem() may release the lock, causing the allocated area
	 * to be unavailable, so try to allocate a free area again.
	 */
	if (mas_nomem(&mas, gfp))
		goto retry;

	if (mas_is_err(&mas))
		ret = xa_err(mas.node);
	else
		*startp = mas.index;

unlock:
	mtree_unlock(mt);
	mas_destroy(&mas);
	return ret;
}
EXPORT_SYMBOL(mtree_alloc_rrange);

/**
 * mtree_erase() - Find an index and erase the entire range.
 * @mt: The maple tree
 * @index: The index to erase
 *
 * Erasing is the same as a walk to an entry then a store of a NULL to that
 * ENTIRE range.  In fact, it is implemented as such using the advanced API.
 *
 * Return: The entry stored at the @index or %NULL
 */
void *mtree_erase(struct maple_tree *mt, unsigned long index)
{
	void *entry = NULL;

	MA_STATE(mas, mt, index, index);
	trace_ma_op(TP_FCT, &mas);

	mtree_lock(mt);
	entry = mas_erase(&mas);
	mtree_unlock(mt);

	return entry;
}
EXPORT_SYMBOL(mtree_erase);

/*
 * mas_dup_free() - Free an incomplete duplication of a tree.
 * @mas: The maple state of a incomplete tree.
 *
 * The parameter @mas->node passed in indicates that the allocation failed on
 * this node. This function frees all nodes starting from @mas->node in the
 * reverse order of mas_dup_build(). There is no need to hold the source tree
 * lock at this time.
 */
static void mas_dup_free(struct ma_state *mas)
{
	struct maple_node *node;
	enum maple_type type;
	void __rcu **slots;
	unsigned char count, i;

	/* Maybe the first node allocation failed. */
	if (mas_is_none(mas))
		return;

	while (!mte_is_root(mas->node)) {
		mas_ascend(mas);
		if (mas->offset) {
			mas->offset--;
			do {
				mas_descend(mas);
				mas->offset = mas_data_end(mas);
			} while (!mte_is_leaf(mas->node));

			mas_ascend(mas);
		}

		node = mte_to_node(mas->node);
		type = mte_node_type(mas->node);
		slots = ma_slots(node, type);
		count = mas_data_end(mas) + 1;
		for (i = 0; i < count; i++)
			((unsigned long *)slots)[i] &= ~MAPLE_NODE_MASK;
		mt_free_bulk(count, slots);
	}

	node = mte_to_node(mas->node);
	kfree(node);
}

/*
 * mas_copy_node() - Copy a maple node and replace the parent.
 * @mas: The maple state of source tree.
 * @new_mas: The maple state of new tree.
 * @parent: The parent of the new node.
 *
 * Copy @mas->node to @new_mas->node, set @parent to be the parent of
 * @new_mas->node. If memory allocation fails, @mas is set to -ENOMEM.
 */
static inline void mas_copy_node(struct ma_state *mas, struct ma_state *new_mas,
		struct maple_pnode *parent)
{
	struct maple_node *node = mte_to_node(mas->node);
	struct maple_node *new_node = mte_to_node(new_mas->node);
	unsigned long val;

	/* Copy the node completely. */
	memcpy(new_node, node, sizeof(struct maple_node));
	/* Update the parent node pointer. */
	val = (unsigned long)node->parent & MAPLE_NODE_MASK;
	new_node->parent = ma_parent_ptr(val | (unsigned long)parent);
}

/*
 * mas_dup_alloc() - Allocate child nodes for a maple node.
 * @mas: The maple state of source tree.
 * @new_mas: The maple state of new tree.
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * This function allocates child nodes for @new_mas->node during the duplication
 * process. If memory allocation fails, @mas is set to -ENOMEM.
 */
static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
		gfp_t gfp)
{
	struct maple_node *node = mte_to_node(mas->node);
	struct maple_node *new_node = mte_to_node(new_mas->node);
	enum maple_type type;
	unsigned char count, i;
	void __rcu **slots;
	void __rcu **new_slots;
	unsigned long val;

	/* Allocate memory for child nodes. */
	type = mte_node_type(mas->node);
	new_slots = ma_slots(new_node, type);
	count = mas->node_request = mas_data_end(mas) + 1;
	mas_alloc_nodes(mas, gfp);
	if (unlikely(mas_is_err(mas)))
		return;

	slots = ma_slots(node, type);
	for (i = 0; i < count; i++) {
		val = (unsigned long)mt_slot_locked(mas->tree, slots, i);
		val &= MAPLE_NODE_MASK;
		new_slots[i] = ma_mnode_ptr((unsigned long)mas_pop_node(mas) |
					    val);
	}
}

/*
 * mas_dup_build() - Build a new maple tree from a source tree
 * @mas: The maple state of source tree, need to be in MAS_START state.
 * @new_mas: The maple state of new tree, need to be in MAS_START state.
 * @gfp: The GFP_FLAGS to use for allocations.
 *
 * This function builds a new tree in DFS preorder. If the memory allocation
 * fails, the error code -ENOMEM will be set in @mas, and @new_mas points to the
 * last node. mas_dup_free() will free the incomplete duplication of a tree.
 *
 * Note that the attributes of the two trees need to be exactly the same, and the
 * new tree needs to be empty, otherwise -EINVAL will be set in @mas.
 */
static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
		gfp_t gfp)
{
	struct maple_node *node;
	struct maple_pnode *parent = NULL;
	struct maple_enode *root;
	enum maple_type type;

	if (unlikely(mt_attr(mas->tree) != mt_attr(new_mas->tree)) ||
	    unlikely(!mtree_empty(new_mas->tree))) {
		mas_set_err(mas, -EINVAL);
		return;
	}

	root = mas_start(mas);
	if (mas_is_ptr(mas) || mas_is_none(mas))
		goto set_new_tree;

	node = mt_alloc_one(gfp);
	if (!node) {
		new_mas->status = ma_none;
		mas_set_err(mas, -ENOMEM);
		return;
	}

	type = mte_node_type(mas->node);
	root = mt_mk_node(node, type);
	new_mas->node = root;
	new_mas->min = 0;
	new_mas->max = ULONG_MAX;
	root = mte_mk_root(root);
	while (1) {
		mas_copy_node(mas, new_mas, parent);
		if (!mte_is_leaf(mas->node)) {
			/* Only allocate child nodes for non-leaf nodes. */
			mas_dup_alloc(mas, new_mas, gfp);
			if (unlikely(mas_is_err(mas)))
				goto empty_mas;
		} else {
			/*
			 * This is the last leaf node and duplication is
			 * completed.
			 */
			if (mas->max == ULONG_MAX)
				goto done;

			/* This is not the last leaf node and needs to go up. */
			do {
				mas_ascend(mas);
				mas_ascend(new_mas);
			} while (mas->offset == mas_data_end(mas));

			/* Move to the next subtree. */
			mas->offset++;
			new_mas->offset++;
		}

		mas_descend(mas);
		parent = ma_parent_ptr(mte_to_node(new_mas->node));
		mas_descend(new_mas);
		mas->offset = 0;
		new_mas->offset = 0;
	}
done:
	/* Specially handle the parent of the root node. */
	mte_to_node(root)->parent = ma_parent_ptr(mas_tree_parent(new_mas));
set_new_tree:
	/* Make them the same height */
	new_mas->tree->ma_flags = mas->tree->ma_flags;
	rcu_assign_pointer(new_mas->tree->ma_root, root);
empty_mas:
	mas_empty_nodes(mas);
}

/**
 * __mt_dup(): Duplicate an entire maple tree
 * @mt: The source maple tree
 * @new: The new maple tree
 * @gfp: The GFP_FLAGS to use for allocations
 *
 * This function duplicates a maple tree in Depth-First Search (DFS) pre-order
 * traversal. It uses memcpy() to copy nodes in the source tree and allocate
 * new child nodes in non-leaf nodes. The new node is exactly the same as the
 * source node except for all the addresses stored in it. It will be faster than
 * traversing all elements in the source tree and inserting them one by one into
 * the new tree.
 * The user needs to ensure that the attributes of the source tree and the new
 * tree are the same, and the new tree needs to be an empty tree, otherwise
 * -EINVAL will be returned.
 * Note that the user needs to manually lock the source tree and the new tree.
 *
 * Return: 0 on success, -ENOMEM if memory could not be allocated, -EINVAL If
 * the attributes of the two trees are different or the new tree is not an empty
 * tree.
 */
int __mt_dup(struct maple_tree *mt, struct maple_tree *new, gfp_t gfp)
{
	int ret = 0;
	MA_STATE(mas, mt, 0, 0);
	MA_STATE(new_mas, new, 0, 0);

	mas_dup_build(&mas, &new_mas, gfp);
	if (unlikely(mas_is_err(&mas))) {
		ret = xa_err(mas.node);
		if (ret == -ENOMEM)
			mas_dup_free(&new_mas);
	}

	return ret;
}
EXPORT_SYMBOL(__mt_dup);

/**
 * mtree_dup(): Duplicate an entire maple tree
 * @mt: The source maple tree
 * @new: The new maple tree
 * @gfp: The GFP_FLAGS to use for allocations
 *
 * This function duplicates a maple tree in Depth-First Search (DFS) pre-order
 * traversal. It uses memcpy() to copy nodes in the source tree and allocate
 * new child nodes in non-leaf nodes. The new node is exactly the same as the
 * source node except for all the addresses stored in it. It will be faster than
 * traversing all elements in the source tree and inserting them one by one into
 * the new tree.
 * The user needs to ensure that the attributes of the source tree and the new
 * tree are the same, and the new tree needs to be an empty tree, otherwise
 * -EINVAL will be returned.
 *
 * Return: 0 on success, -ENOMEM if memory could not be allocated, -EINVAL If
 * the attributes of the two trees are different or the new tree is not an empty
 * tree.
 */
int mtree_dup(struct maple_tree *mt, struct maple_tree *new, gfp_t gfp)
{
	int ret = 0;
	MA_STATE(mas, mt, 0, 0);
	MA_STATE(new_mas, new, 0, 0);

	mas_lock(&new_mas);
	mas_lock_nested(&mas, SINGLE_DEPTH_NESTING);
	mas_dup_build(&mas, &new_mas, gfp);
	mas_unlock(&mas);
	if (unlikely(mas_is_err(&mas))) {
		ret = xa_err(mas.node);
		if (ret == -ENOMEM)
			mas_dup_free(&new_mas);
	}

	mas_unlock(&new_mas);
	return ret;
}
EXPORT_SYMBOL(mtree_dup);

/**
 * __mt_destroy() - Walk and free all nodes of a locked maple tree.
 * @mt: The maple tree
 *
 * Note: Does not handle locking.
 */
void __mt_destroy(struct maple_tree *mt)
{
	void *root = mt_root_locked(mt);

	rcu_assign_pointer(mt->ma_root, NULL);
	if (xa_is_node(root))
		mte_destroy_walk(root, mt);

	mt->ma_flags = mt_attr(mt);
}
EXPORT_SYMBOL_GPL(__mt_destroy);

/**
 * mtree_destroy() - Destroy a maple tree
 * @mt: The maple tree
 *
 * Frees all resources used by the tree.  Handles locking.
 */
void mtree_destroy(struct maple_tree *mt)
{
	mtree_lock(mt);
	__mt_destroy(mt);
	mtree_unlock(mt);
}
EXPORT_SYMBOL(mtree_destroy);

/**
 * mt_find() - Search from the start up until an entry is found.
 * @mt: The maple tree
 * @index: Pointer which contains the start location of the search
 * @max: The maximum value of the search range
 *
 * Takes RCU read lock internally to protect the search, which does not
 * protect the returned pointer after dropping RCU read lock.
 * See also: Documentation/core-api/maple_tree.rst
 *
 * In case that an entry is found @index is updated to point to the next
 * possible entry independent whether the found entry is occupying a
 * single index or a range if indices.
 *
 * Return: The entry at or after the @index or %NULL
 */
void *mt_find(struct maple_tree *mt, unsigned long *index, unsigned long max)
{
	MA_STATE(mas, mt, *index, *index);
	void *entry;
#ifdef CONFIG_DEBUG_MAPLE_TREE
	unsigned long copy = *index;
#endif

	trace_ma_read(TP_FCT, &mas);

	if ((*index) > max)
		return NULL;

	rcu_read_lock();
retry:
	entry = mas_state_walk(&mas);
	if (mas_is_start(&mas))
		goto retry;

	if (unlikely(xa_is_zero(entry)))
		entry = NULL;

	if (entry)
		goto unlock;

	while (mas_is_active(&mas) && (mas.last < max)) {
		entry = mas_next_slot(&mas, max, false);
		if (likely(entry && !xa_is_zero(entry)))
			break;
	}

	if (unlikely(xa_is_zero(entry)))
		entry = NULL;
unlock:
	rcu_read_unlock();
	if (likely(entry)) {
		*index = mas.last + 1;
#ifdef CONFIG_DEBUG_MAPLE_TREE
		if (MT_WARN_ON(mt, (*index) && ((*index) <= copy)))
			pr_err("index not increased! %lx <= %lx\n",
			       *index, copy);
#endif
	}

	return entry;
}
EXPORT_SYMBOL(mt_find);

/**
 * mt_find_after() - Search from the start up until an entry is found.
 * @mt: The maple tree
 * @index: Pointer which contains the start location of the search
 * @max: The maximum value to check
 *
 * Same as mt_find() except that it checks @index for 0 before
 * searching. If @index == 0, the search is aborted. This covers a wrap
 * around of @index to 0 in an iterator loop.
 *
 * Return: The entry at or after the @index or %NULL
 */
void *mt_find_after(struct maple_tree *mt, unsigned long *index,
		    unsigned long max)
{
	if (!(*index))
		return NULL;

	return mt_find(mt, index, max);
}
EXPORT_SYMBOL(mt_find_after);

#ifdef CONFIG_DEBUG_MAPLE_TREE
atomic_t maple_tree_tests_run;
EXPORT_SYMBOL_GPL(maple_tree_tests_run);
atomic_t maple_tree_tests_passed;
EXPORT_SYMBOL_GPL(maple_tree_tests_passed);

#ifndef __KERNEL__
extern void kmem_cache_set_non_kernel(struct kmem_cache *, unsigned int);
void mt_set_non_kernel(unsigned int val)
{
	kmem_cache_set_non_kernel(maple_node_cache, val);
}

extern void kmem_cache_set_callback(struct kmem_cache *cachep,
		void (*callback)(void *));
void mt_set_callback(void (*callback)(void *))
{
	kmem_cache_set_callback(maple_node_cache, callback);
}

extern void kmem_cache_set_private(struct kmem_cache *cachep, void *private);
void mt_set_private(void *private)
{
	kmem_cache_set_private(maple_node_cache, private);
}

extern unsigned long kmem_cache_get_alloc(struct kmem_cache *);
unsigned long mt_get_alloc_size(void)
{
	return kmem_cache_get_alloc(maple_node_cache);
}

extern void kmem_cache_zero_nr_tallocated(struct kmem_cache *);
void mt_zero_nr_tallocated(void)
{
	kmem_cache_zero_nr_tallocated(maple_node_cache);
}

extern unsigned int kmem_cache_nr_tallocated(struct kmem_cache *);
unsigned int mt_nr_tallocated(void)
{
	return kmem_cache_nr_tallocated(maple_node_cache);
}

extern unsigned int kmem_cache_nr_allocated(struct kmem_cache *);
unsigned int mt_nr_allocated(void)
{
	return kmem_cache_nr_allocated(maple_node_cache);
}

void mt_cache_shrink(void)
{
}
#else
/*
 * mt_cache_shrink() - For testing, don't use this.
 *
 * Certain testcases can trigger an OOM when combined with other memory
 * debugging configuration options.  This function is used to reduce the
 * possibility of an out of memory even due to kmem_cache objects remaining
 * around for longer than usual.
 */
void mt_cache_shrink(void)
{
	kmem_cache_shrink(maple_node_cache);

}
EXPORT_SYMBOL_GPL(mt_cache_shrink);

#endif /* not defined __KERNEL__ */
/*
 * mas_get_slot() - Get the entry in the maple state node stored at @offset.
 * @mas: The maple state
 * @offset: The offset into the slot array to fetch.
 *
 * Return: The entry stored at @offset.
 */
static inline struct maple_enode *mas_get_slot(struct ma_state *mas,
		unsigned char offset)
{
	return mas_slot(mas, ma_slots(mas_mn(mas), mte_node_type(mas->node)),
			offset);
}

/* Depth first search, post-order */
static void mas_dfs_postorder(struct ma_state *mas, unsigned long max)
{

	struct maple_enode *p, *mn = mas->node;
	unsigned long p_min, p_max;

	mas_next_node(mas, mas_mn(mas), max);
	if (!mas_is_overflow(mas))
		return;

	if (mte_is_root(mn))
		return;

	mas->node = mn;
	mas_ascend(mas);
	do {
		p = mas->node;
		p_min = mas->min;
		p_max = mas->max;
		mas_prev_node(mas, 0);
	} while (!mas_is_underflow(mas));

	mas->node = p;
	mas->max = p_max;
	mas->min = p_min;
}

/* Tree validations */
static void mt_dump_node(const struct maple_tree *mt, void *entry,
		unsigned long min, unsigned long max, unsigned int depth,
		enum mt_dump_format format);
static void mt_dump_range(unsigned long min, unsigned long max,
			  unsigned int depth, enum mt_dump_format format)
{
	static const char spaces[] = "                                ";

	switch(format) {
	case mt_dump_hex:
		if (min == max)
			pr_info("%.*s%lx: ", depth * 2, spaces, min);
		else
			pr_info("%.*s%lx-%lx: ", depth * 2, spaces, min, max);
		break;
	case mt_dump_dec:
		if (min == max)
			pr_info("%.*s%lu: ", depth * 2, spaces, min);
		else
			pr_info("%.*s%lu-%lu: ", depth * 2, spaces, min, max);
	}
}

static void mt_dump_entry(void *entry, unsigned long min, unsigned long max,
			  unsigned int depth, enum mt_dump_format format)
{
	mt_dump_range(min, max, depth, format);

	if (xa_is_value(entry))
		pr_cont("value %ld (0x%lx) [" PTR_FMT "]\n", xa_to_value(entry),
			xa_to_value(entry), entry);
	else if (xa_is_zero(entry))
		pr_cont("zero (%ld)\n", xa_to_internal(entry));
	else if (mt_is_reserved(entry))
		pr_cont("UNKNOWN ENTRY (" PTR_FMT ")\n", entry);
	else
		pr_cont(PTR_FMT "\n", entry);
}

static void mt_dump_range64(const struct maple_tree *mt, void *entry,
		unsigned long min, unsigned long max, unsigned int depth,
		enum mt_dump_format format)
{
	struct maple_range_64 *node = &mte_to_node(entry)->mr64;
	bool leaf = mte_is_leaf(entry);
	unsigned long first = min;
	int i;

	pr_cont(" contents: ");
	for (i = 0; i < MAPLE_RANGE64_SLOTS - 1; i++) {
		switch(format) {
		case mt_dump_hex:
			pr_cont(PTR_FMT " %lX ", node->slot[i], node->pivot[i]);
			break;
		case mt_dump_dec:
			pr_cont(PTR_FMT " %lu ", node->slot[i], node->pivot[i]);
		}
	}
	pr_cont(PTR_FMT "\n", node->slot[i]);
	for (i = 0; i < MAPLE_RANGE64_SLOTS; i++) {
		unsigned long last = max;

		if (i < (MAPLE_RANGE64_SLOTS - 1))
			last = node->pivot[i];
		else if (!node->slot[i] && max != mt_node_max(entry))
			break;
		if (last == 0 && i > 0)
			break;
		if (leaf)
			mt_dump_entry(mt_slot(mt, node->slot, i),
					first, last, depth + 1, format);
		else if (node->slot[i])
			mt_dump_node(mt, mt_slot(mt, node->slot, i),
					first, last, depth + 1, format);

		if (last == max)
			break;
		if (last > max) {
			switch(format) {
			case mt_dump_hex:
				pr_err("node " PTR_FMT " last (%lx) > max (%lx) at pivot %d!\n",
					node, last, max, i);
				break;
			case mt_dump_dec:
				pr_err("node " PTR_FMT " last (%lu) > max (%lu) at pivot %d!\n",
					node, last, max, i);
			}
		}
		first = last + 1;
	}
}

static void mt_dump_arange64(const struct maple_tree *mt, void *entry,
	unsigned long min, unsigned long max, unsigned int depth,
	enum mt_dump_format format)
{
	struct maple_arange_64 *node = &mte_to_node(entry)->ma64;
	unsigned long first = min;
	int i;

	pr_cont(" contents: ");
	for (i = 0; i < MAPLE_ARANGE64_SLOTS; i++) {
		switch (format) {
		case mt_dump_hex:
			pr_cont("%lx ", node->gap[i]);
			break;
		case mt_dump_dec:
			pr_cont("%lu ", node->gap[i]);
		}
	}
	pr_cont("| %02X %02X| ", node->meta.end, node->meta.gap);
	for (i = 0; i < MAPLE_ARANGE64_SLOTS - 1; i++) {
		switch (format) {
		case mt_dump_hex:
			pr_cont(PTR_FMT " %lX ", node->slot[i], node->pivot[i]);
			break;
		case mt_dump_dec:
			pr_cont(PTR_FMT " %lu ", node->slot[i], node->pivot[i]);
		}
	}
	pr_cont(PTR_FMT "\n", node->slot[i]);
	for (i = 0; i < MAPLE_ARANGE64_SLOTS; i++) {
		unsigned long last = max;

		if (i < (MAPLE_ARANGE64_SLOTS - 1))
			last = node->pivot[i];
		else if (!node->slot[i])
			break;
		if (last == 0 && i > 0)
			break;
		if (node->slot[i])
			mt_dump_node(mt, mt_slot(mt, node->slot, i),
					first, last, depth + 1, format);

		if (last == max)
			break;
		if (last > max) {
			switch(format) {
			case mt_dump_hex:
				pr_err("node " PTR_FMT " last (%lx) > max (%lx) at pivot %d!\n",
					node, last, max, i);
				break;
			case mt_dump_dec:
				pr_err("node " PTR_FMT " last (%lu) > max (%lu) at pivot %d!\n",
					node, last, max, i);
			}
		}
		first = last + 1;
	}
}

static void mt_dump_node(const struct maple_tree *mt, void *entry,
		unsigned long min, unsigned long max, unsigned int depth,
		enum mt_dump_format format)
{
	struct maple_node *node = mte_to_node(entry);
	unsigned int type = mte_node_type(entry);
	unsigned int i;

	mt_dump_range(min, max, depth, format);

	pr_cont("node " PTR_FMT " depth %d type %d parent " PTR_FMT, node,
		depth, type, node ? node->parent : NULL);
	switch (type) {
	case maple_dense:
		pr_cont("\n");
		for (i = 0; i < MAPLE_NODE_SLOTS; i++) {
			if (min + i > max)
				pr_cont("OUT OF RANGE: ");
			mt_dump_entry(mt_slot(mt, node->slot, i),
					min + i, min + i, depth, format);
		}
		break;
	case maple_leaf_64:
	case maple_range_64:
		mt_dump_range64(mt, entry, min, max, depth, format);
		break;
	case maple_arange_64:
		mt_dump_arange64(mt, entry, min, max, depth, format);
		break;

	default:
		pr_cont(" UNKNOWN TYPE\n");
	}
}

void mt_dump(const struct maple_tree *mt, enum mt_dump_format format)
{
	void *entry = rcu_dereference_check(mt->ma_root, mt_locked(mt));

	pr_info("maple_tree(" PTR_FMT ") flags %X, height %u root " PTR_FMT "\n",
		 mt, mt->ma_flags, mt_height(mt), entry);
	if (xa_is_node(entry))
		mt_dump_node(mt, entry, 0, mt_node_max(entry), 0, format);
	else if (entry)
		mt_dump_entry(entry, 0, 0, 0, format);
	else
		pr_info("(empty)\n");
}
EXPORT_SYMBOL_GPL(mt_dump);

/*
 * Calculate the maximum gap in a node and check if that's what is reported in
 * the parent (unless root).
 */
static void mas_validate_gaps(struct ma_state *mas)
{
	struct maple_enode *mte = mas->node;
	struct maple_node *p_mn, *node = mte_to_node(mte);
	enum maple_type mt = mte_node_type(mas->node);
	unsigned long gap = 0, max_gap = 0;
	unsigned long p_end, p_start = mas->min;
	unsigned char p_slot, offset;
	unsigned long *gaps = NULL;
	unsigned long *pivots = ma_pivots(node, mt);
	unsigned int i;

	if (ma_is_dense(mt)) {
		for (i = 0; i < mt_slot_count(mte); i++) {
			if (mas_get_slot(mas, i)) {
				if (gap > max_gap)
					max_gap = gap;
				gap = 0;
				continue;
			}
			gap++;
		}
		goto counted;
	}

	gaps = ma_gaps(node, mt);
	for (i = 0; i < mt_slot_count(mte); i++) {
		p_end = mas_safe_pivot(mas, pivots, i, mt);

		if (!gaps) {
			if (!mas_get_slot(mas, i))
				gap = p_end - p_start + 1;
		} else {
			void *entry = mas_get_slot(mas, i);

			gap = gaps[i];
			MT_BUG_ON(mas->tree, !entry);

			if (gap > p_end - p_start + 1) {
				pr_err(PTR_FMT "[%u] %lu >= %lu - %lu + 1 (%lu)\n",
				       mas_mn(mas), i, gap, p_end, p_start,
				       p_end - p_start + 1);
				MT_BUG_ON(mas->tree, gap > p_end - p_start + 1);
			}
		}

		if (gap > max_gap)
			max_gap = gap;

		p_start = p_end + 1;
		if (p_end >= mas->max)
			break;
	}

counted:
	if (mt == maple_arange_64) {
		MT_BUG_ON(mas->tree, !gaps);
		offset = ma_meta_gap(node);
		if (offset > i) {
			pr_err("gap offset " PTR_FMT "[%u] is invalid\n", node, offset);
			MT_BUG_ON(mas->tree, 1);
		}

		if (gaps[offset] != max_gap) {
			pr_err("gap " PTR_FMT "[%u] is not the largest gap %lu\n",
			       node, offset, max_gap);
			MT_BUG_ON(mas->tree, 1);
		}

		for (i++ ; i < mt_slot_count(mte); i++) {
			if (gaps[i] != 0) {
				pr_err("gap " PTR_FMT "[%u] beyond node limit != 0\n",
				       node, i);
				MT_BUG_ON(mas->tree, 1);
			}
		}
	}

	if (mte_is_root(mte))
		return;

	p_slot = mte_parent_slot(mas->node);
	p_mn = mte_parent(mte);
	MT_BUG_ON(mas->tree, max_gap > mas->max);
	if (ma_gaps(p_mn, mas_parent_type(mas, mte))[p_slot] != max_gap) {
		pr_err("gap " PTR_FMT "[%u] != %lu\n", p_mn, p_slot, max_gap);
		mt_dump(mas->tree, mt_dump_hex);
		MT_BUG_ON(mas->tree, 1);
	}
}

static void mas_validate_parent_slot(struct ma_state *mas)
{
	struct maple_node *parent;
	struct maple_enode *node;
	enum maple_type p_type;
	unsigned char p_slot;
	void __rcu **slots;
	int i;

	if (mte_is_root(mas->node))
		return;

	p_slot = mte_parent_slot(mas->node);
	p_type = mas_parent_type(mas, mas->node);
	parent = mte_parent(mas->node);
	slots = ma_slots(parent, p_type);
	MT_BUG_ON(mas->tree, mas_mn(mas) == parent);

	/* Check prev/next parent slot for duplicate node entry */

	for (i = 0; i < mt_slots[p_type]; i++) {
		node = mas_slot(mas, slots, i);
		if (i == p_slot) {
			if (node != mas->node)
				pr_err("parent " PTR_FMT "[%u] does not have " PTR_FMT "\n",
					parent, i, mas_mn(mas));
			MT_BUG_ON(mas->tree, node != mas->node);
		} else if (node == mas->node) {
			pr_err("Invalid child " PTR_FMT " at parent " PTR_FMT "[%u] p_slot %u\n",
			       mas_mn(mas), parent, i, p_slot);
			MT_BUG_ON(mas->tree, node == mas->node);
		}
	}
}

static void mas_validate_child_slot(struct ma_state *mas)
{
	enum maple_type type = mte_node_type(mas->node);
	void __rcu **slots = ma_slots(mte_to_node(mas->node), type);
	unsigned long *pivots = ma_pivots(mte_to_node(mas->node), type);
	struct maple_enode *child;
	unsigned char i;

	if (mte_is_leaf(mas->node))
		return;

	for (i = 0; i < mt_slots[type]; i++) {
		child = mas_slot(mas, slots, i);

		if (!child) {
			pr_err("Non-leaf node lacks child at " PTR_FMT "[%u]\n",
			       mas_mn(mas), i);
			MT_BUG_ON(mas->tree, 1);
		}

		if (mte_parent_slot(child) != i) {
			pr_err("Slot error at " PTR_FMT "[%u]: child " PTR_FMT " has pslot %u\n",
			       mas_mn(mas), i, mte_to_node(child),
			       mte_parent_slot(child));
			MT_BUG_ON(mas->tree, 1);
		}

		if (mte_parent(child) != mte_to_node(mas->node)) {
			pr_err("child " PTR_FMT " has parent " PTR_FMT " not " PTR_FMT "\n",
			       mte_to_node(child), mte_parent(child),
			       mte_to_node(mas->node));
			MT_BUG_ON(mas->tree, 1);
		}

		if (i < mt_pivots[type] && pivots[i] == mas->max)
			break;
	}
}

/*
 * Validate all pivots are within mas->min and mas->max, check metadata ends
 * where the maximum ends and ensure there is no slots or pivots set outside of
 * the end of the data.
 */
static void mas_validate_limits(struct ma_state *mas)
{
	int i;
	unsigned long prev_piv = 0;
	enum maple_type type = mte_node_type(mas->node);
	void __rcu **slots = ma_slots(mte_to_node(mas->node), type);
	unsigned long *pivots = ma_pivots(mas_mn(mas), type);

	for (i = 0; i < mt_slots[type]; i++) {
		unsigned long piv;

		piv = mas_safe_pivot(mas, pivots, i, type);

		if (!piv && (i != 0)) {
			pr_err("Missing node limit pivot at " PTR_FMT "[%u]",
			       mas_mn(mas), i);
			MAS_WARN_ON(mas, 1);
		}

		if (prev_piv > piv) {
			pr_err(PTR_FMT "[%u] piv %lu < prev_piv %lu\n",
				mas_mn(mas), i, piv, prev_piv);
			MAS_WARN_ON(mas, piv < prev_piv);
		}

		if (piv < mas->min) {
			pr_err(PTR_FMT "[%u] %lu < %lu\n", mas_mn(mas), i,
				piv, mas->min);
			MAS_WARN_ON(mas, piv < mas->min);
		}
		if (piv > mas->max) {
			pr_err(PTR_FMT "[%u] %lu > %lu\n", mas_mn(mas), i,
				piv, mas->max);
			MAS_WARN_ON(mas, piv > mas->max);
		}
		prev_piv = piv;
		if (piv == mas->max)
			break;
	}

	if (mas_data_end(mas) != i) {
		pr_err("node" PTR_FMT ": data_end %u != the last slot offset %u\n",
		       mas_mn(mas), mas_data_end(mas), i);
		MT_BUG_ON(mas->tree, 1);
	}

	for (i += 1; i < mt_slots[type]; i++) {
		void *entry = mas_slot(mas, slots, i);

		if (entry && (i != mt_slots[type] - 1)) {
			pr_err(PTR_FMT "[%u] should not have entry " PTR_FMT "\n",
			       mas_mn(mas), i, entry);
			MT_BUG_ON(mas->tree, entry != NULL);
		}

		if (i < mt_pivots[type]) {
			unsigned long piv = pivots[i];

			if (!piv)
				continue;

			pr_err(PTR_FMT "[%u] should not have piv %lu\n",
			       mas_mn(mas), i, piv);
			MAS_WARN_ON(mas, i < mt_pivots[type] - 1);
		}
	}
}

static void mt_validate_nulls(struct maple_tree *mt)
{
	void *entry, *last = (void *)1;
	unsigned char offset = 0;
	void __rcu **slots;
	MA_STATE(mas, mt, 0, 0);

	mas_start(&mas);
	if (mas_is_none(&mas) || (mas_is_ptr(&mas)))
		return;

	while (!mte_is_leaf(mas.node))
		mas_descend(&mas);

	slots = ma_slots(mte_to_node(mas.node), mte_node_type(mas.node));
	do {
		entry = mas_slot(&mas, slots, offset);
		if (!last && !entry) {
			pr_err("Sequential nulls end at " PTR_FMT "[%u]\n",
				mas_mn(&mas), offset);
		}
		MT_BUG_ON(mt, !last && !entry);
		last = entry;
		if (offset == mas_data_end(&mas)) {
			mas_next_node(&mas, mas_mn(&mas), ULONG_MAX);
			if (mas_is_overflow(&mas))
				return;
			offset = 0;
			slots = ma_slots(mte_to_node(mas.node),
					 mte_node_type(mas.node));
		} else {
			offset++;
		}

	} while (!mas_is_overflow(&mas));
}

/*
 * validate a maple tree by checking:
 * 1. The limits (pivots are within mas->min to mas->max)
 * 2. The gap is correctly set in the parents
 */
void mt_validate(struct maple_tree *mt)
	__must_hold(mas->tree->ma_lock)
{
	unsigned char end;

	MA_STATE(mas, mt, 0, 0);
	mas_start(&mas);
	if (!mas_is_active(&mas))
		return;

	while (!mte_is_leaf(mas.node))
		mas_descend(&mas);

	while (!mas_is_overflow(&mas)) {
		MAS_WARN_ON(&mas, mte_dead_node(mas.node));
		end = mas_data_end(&mas);
		if (MAS_WARN_ON(&mas, (end < mt_min_slot_count(mas.node)) &&
				(!mte_is_root(mas.node)))) {
			pr_err("Invalid size %u of " PTR_FMT "\n",
			       end, mas_mn(&mas));
		}

		mas_validate_parent_slot(&mas);
		mas_validate_limits(&mas);
		mas_validate_child_slot(&mas);
		if (mt_is_alloc(mt))
			mas_validate_gaps(&mas);
		mas_dfs_postorder(&mas, ULONG_MAX);
	}
	mt_validate_nulls(mt);
}
EXPORT_SYMBOL_GPL(mt_validate);

void mas_dump(const struct ma_state *mas)
{
	pr_err("MAS: tree=" PTR_FMT " enode=" PTR_FMT " ",
	       mas->tree, mas->node);
	switch (mas->status) {
	case ma_active:
		pr_err("(ma_active)");
		break;
	case ma_none:
		pr_err("(ma_none)");
		break;
	case ma_root:
		pr_err("(ma_root)");
		break;
	case ma_start:
		pr_err("(ma_start) ");
		break;
	case ma_pause:
		pr_err("(ma_pause) ");
		break;
	case ma_overflow:
		pr_err("(ma_overflow) ");
		break;
	case ma_underflow:
		pr_err("(ma_underflow) ");
		break;
	case ma_error:
		pr_err("(ma_error) ");
		break;
	}

	pr_err("Store Type: ");
	switch (mas->store_type) {
	case wr_invalid:
		pr_err("invalid store type\n");
		break;
	case wr_new_root:
		pr_err("new_root\n");
		break;
	case wr_store_root:
		pr_err("store_root\n");
		break;
	case wr_exact_fit:
		pr_err("exact_fit\n");
		break;
	case wr_split_store:
		pr_err("split_store\n");
		break;
	case wr_slot_store:
		pr_err("slot_store\n");
		break;
	case wr_append:
		pr_err("append\n");
		break;
	case wr_node_store:
		pr_err("node_store\n");
		break;
	case wr_spanning_store:
		pr_err("spanning_store\n");
		break;
	case wr_rebalance:
		pr_err("rebalance\n");
		break;
	}

	pr_err("[%u/%u] index=%lx last=%lx\n", mas->offset, mas->end,
	       mas->index, mas->last);
	pr_err("     min=%lx max=%lx sheaf=" PTR_FMT ", request %lu depth=%u, flags=%x\n",
	       mas->min, mas->max, mas->sheaf, mas->node_request, mas->depth,
	       mas->mas_flags);
	if (mas->index > mas->last)
		pr_err("Check index & last\n");
}
EXPORT_SYMBOL_GPL(mas_dump);

void mas_wr_dump(const struct ma_wr_state *wr_mas)
{
	pr_err("WR_MAS: node=" PTR_FMT " r_min=%lx r_max=%lx\n",
	       wr_mas->node, wr_mas->r_min, wr_mas->r_max);
	pr_err("        type=%u off_end=%u, node_end=%u, end_piv=%lx\n",
	       wr_mas->type, wr_mas->offset_end, wr_mas->mas->end,
	       wr_mas->end_piv);
}
EXPORT_SYMBOL_GPL(mas_wr_dump);

#endif /* CONFIG_DEBUG_MAPLE_TREE */
]

gemini-3-flash-preview

							input: 141608
							output: 39
							thoughts: 4786
						

**Analysis of Data Race in Maple Tree 'parent' Pointer**

Okay, I've thoroughly examined this KCSAN report concerning a data race on the `parent` pointer within the Maple Tree implementation. Initially, I didn't see explicit `READ_ONCE` or `WRITE_ONCE` usage, which immediately raised a flag. However, the presence of `rcu_assign_pointer` and `rcu_dereference_*` hinted at RCU semantics, albeit not in the typical access pattern for the `parent` field itself, which is a `struct maple_pnode *`.

The key lies in how `mte_set_node_dead` and `ma_dead_node` interact.  `mte_set_node_dead` sets a node's `parent` pointer to itself, effectively marking it as "dead," which is crucial for RCU-based node removal.  The accompanying `smp_wmb()` is vital here. *After* the pointer write, it ensures that this state change is visible to other CPUs. I initially thought this was unusual, as such barriers typically come *before* data writes. The rationale is to ensure the "dead" state is propagated before freeing the node. `ma_dead_node` then reads this `parent` pointer.  The `smp_rmb()` *before* the read ensures that any reads performed prior to checking the "dead" flag are completed, maintaining consistency.

The race itself is real – KCSAN is correct in identifying it. However, the critical question is whether it's *harmful*.  Since pointer writes are atomic on 64-bit systems, there's no risk of tearing. The design appears robust: if a reader encounters the old value, it continues; if the new value, it retries.  Because the node isn't freed until the grace period, using the old pointer is safe. However, there's a more significant concern: the `parent` pointer is used not just for the "dead" flag but also for tree traversal, for example, `mas_ascend`.  If a reader is traversing up the tree (e.g., in `mas_ascend`) and the `parent` pointer is changed mid-traversal to point to the node itself (marking it dead), there is confusion.

By examining `mas_ascend`, I determined this is handled correctly. `mte_parent` reads `node->parent`. If "dead," the condition `a_node == p_node` becomes true, returning 1, which appropriately indicates a dead node. I was initially worried about a torn `parent` pointer and the potential for a crash. Still, on x86_64, aligned 64-bit accesses are atomic, thus eliminating the possibility of tearing. 32-bit pointers are also aligned and atomic.  However, even without tearing, there are still potential race conditions, as the compiler is free to read the parent pointer multiple times and make assumptions about its value.

The code's lack of `READ_ONCE` and `WRITE_ONCE` is a violation of memory model rules, even if the result is correct.  I reviewed the code and determined that the maple tree designers are using explicit retry logic and double-checks to attempt to defend against race conditions. Specifically, the double-check in `mas_ascend` (`if (p_node != mte_parent(mas->node))`) is present to mitigate precisely the kind of race we're discussing.

If the compiler is overly aggressive, the second check can be optimized out. This is where KCSAN's detection becomes vital. The code may be benign in the sense that it doesn't crash a conservative compiler, but the race exposes a violation of the memory model and potential issues.

I then dug deeper, I encountered a report similar to this with the same conclusion – a real bug, requiring `READ_ONCE`. I verified this with Liam Howlett's patch series, which introduces `READ_ONCE` and `WRITE_ONCE` to address such reports. The reports show the transition from "not dead" to "dead." The fact is that this transition may not happen atomically, so it's a real race condition. The fact that the code repeatedly checks the flag mitigates the risk, but does not eliminate it.

I verified the code's resilience by looking at how `mas_state_walk`, `mtree_range_walk`, and `ma_dead_node` interact, and the implementation uses RCU and double-checks at every level. This reinforces that the race is handled gracefully: a stale value means a slightly longer walk, or a retry.

The barriers' roles in `mte_set_node_dead` and `ma_dead_node` ensure data consistency,

map[ContextFile:mm/vma.c IncludeLines:true Name:vma_complete]

map[SourceCode: 327:	/*
 328:	 * vma_complete- Helper function for handling the unlocking after altering VMAs,
 329:	 * or for inserting a VMA.
 330:	 *
 331:	 * @vp: The vma_prepare struct
 332:	 * @vmi: The vma iterator
 333:	 * @mm: The mm_struct
 334:	 */
 335:	static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,
 336:				 struct mm_struct *mm)
 337:	{
 338:		if (vp->file) {
 339:			if (vp->adj_next)
 340:				vma_interval_tree_insert(vp->adj_next,
 341:							 &vp->mapping->i_mmap);
 342:			vma_interval_tree_insert(vp->vma, &vp->mapping->i_mmap);
 343:			flush_dcache_mmap_unlock(vp->mapping);
 344:		}
 345:	
 346:		if (vp->remove && vp->file) {
 347:			__remove_shared_vm_struct(vp->remove, vp->mapping);
 348:			if (vp->remove2)
 349:				__remove_shared_vm_struct(vp->remove2, vp->mapping);
 350:		} else if (vp->insert) {
 351:			/*
 352:			 * split_vma has split insert from vma, and needs
 353:			 * us to insert it before dropping the locks
 354:			 * (it may either follow vma or precede it).
 355:			 */
 356:			vma_iter_store_new(vmi, vp->insert);
 357:			mm->map_count++;
 358:		}
 359:	
 360:		if (vp->anon_vma) {
 361:			anon_vma_interval_tree_post_update_vma(vp->vma);
 362:			if (vp->adj_next)
 363:				anon_vma_interval_tree_post_update_vma(vp->adj_next);
 364:			anon_vma_unlock_write(vp->anon_vma);
 365:		}
 366:	
 367:		if (vp->file) {
 368:			i_mmap_unlock_write(vp->mapping);
 369:	
 370:			if (!vp->skip_vma_uprobe) {
 371:				uprobe_mmap(vp->vma);
 372:	
 373:				if (vp->adj_next)
 374:					uprobe_mmap(vp->adj_next);
 375:			}
 376:		}
 377:	
 378:		if (vp->remove) {
 379:	again:
 380:			vma_mark_detached(vp->remove);
 381:			if (vp->file) {
 382:				uprobe_munmap(vp->remove, vp->remove->vm_start,
 383:					      vp->remove->vm_end);
 384:				fput(vp->file);
 385:			}
 386:			if (vp->remove->anon_vma)
 387:				unlink_anon_vmas(vp->remove);
 388:			mm->map_count--;
 389:			mpol_put(vma_policy(vp->remove));
 390:			if (!vp->remove2)
 391:				WARN_ON_ONCE(vp->vma->vm_end < vp->remove->vm_end);
 392:			vm_area_free(vp->remove);
 393:	
 394:			/*
 395:			 * In mprotect's case 6 (see comments on vma_merge),
 396:			 * we are removing both mid and next vmas
 397:			 */
 398:			if (vp->remove2) {
 399:				vp->remove = vp->remove2;
 400:				vp->remove2 = NULL;
 401:				goto again;
 402:			}
 403:		}
 404:		if (vp->insert && vp->file)
 405:			uprobe_mmap(vp->insert);
 406:	}
 SourceFile:mm/vma.c]

gemini-3-flash-preview

							input: 148704
							output: 19
							thoughts: 479
						

**Analyzing the Potential Race Condition in the VMA Tree**

Okay, so `vma_iter_store_new` eventually calls `mas_store_prealloc`, and we're definitely in RCU mode, which is the key here. The race is pretty clear: a writer's updating the VMA tree while a reader, triggered by a page fault, is doing its thing with `lock_vma_under_rcu`. This is precisely the scenario RCU is designed to handle.

The crucial point is that data race on `node->parent`, and it's a known property of the maple tree's RCU implementation, which is okay because: (1) pointer writes are atomic on the architectures we're talking about, (2) the logic correctly handles those stale values, either retrying or just continuing with consistent data, and (3) memory barriers are used so the "dead" flag and data are properly ordered.

I wanted to make absolutely sure the compiler wasn't doing something funky with the `parent == node` check. Is there any risk that the compiler might "optimize" it in a way that, if it was false once, it would *always* return false? I had to run through that logic a bit. No, that's not possible, because it's a concurrent access; the compiler doesn't know that `node->parent` won't change unless the compiler would hoist the read (which is prevented by the loop logic). And since `ma_dead_node` is called at every level of the walk, with `smp_rmb()` within it, that acts as a compiler barrier.

Hold on a second, that's it! `smp_rmb()` is a compiler barrier on *all* architectures. Okay, so that means the compiler *can't* hoist the read of `node->parent` *across* `smp_rmb()`. But `smp_rmb()` is called *before* the check. So it doesn't prevent hoisting the read to *after* the check, but then, there's nothing after the check in `ma_dead_node`.

Looking at `mtree_range_walk`:

```c
next = mt_slot(mas->tree, slots, offset);
if (unlikely(ma_dead_node(node)))
    goto dead_node;
```

The `smp_rmb()` within `ma_dead_node` is key. It makes sure that `next` is read before the `node->parent` read. The compiler *can't* reorder `node->parent` before `smp_rmb()`. Therefore, the order is guaranteed, and the race is benign.

map[Benign:true Confident:true]

map[Benign:true Confident:true]

gemini-3-flash-preview

							input: 149803
							output: 691
							thoughts: 384
						

**My Assessment of the Maple Tree Data Race**

Okay, so I've been looking at this potential data race in the maple tree code, and here's my take. The core of the problem is between `mas_wmb_replace`, which is writing to `node->parent` to essentially declare a node as "dead," and `mas_state_walk`, which is reading `node->parent` to determine if a node is still valid. This is a pretty standard RCU-style synchronization scheme, and a common pattern in this kind of data structure.

First, let's establish the fundamentals. The `parent` field is a simple pointer, naturally aligned on all our supported architectures. Since pointer-sized loads and stores are atomic, at the hardware level, we're already starting in a reasonably safe place. No worries about partial writes.

Now, concerning the actual logic of the tree. The maple tree is architected to handle concurrent readers using RCU. When we replace a node, we mark it as "dead" by pointing its `parent` to itself. Readers, at several points, check this "dead" flag during their walk. What happens if a reader sees a "stale" value, and the "dead" flag isn't set? No problem, it keeps walking the old node, which is perfectly safe because the contents of that old node are still valid until the end of the RCU grace period. If, however, they see the new "dead" value, they just retry their walk from the root.

The crucial ingredient here is the memory barrier. We're using `smp_wmb()` after marking the node dead and `smp_rmb()` before checking the dead flag. These barriers are the glue that holds everything together. They guarantee that the readers see a consistent state of the tree and the node's contents relative to the dead flag. This memory barrier is also a crucial compiler barrier. Even though we aren't using `READ_ONCE` or `WRITE_ONCE`, these barriers prevent the compiler from performing any nasty reordering that might lead to inconsistencies.

Given all of this, the data race isn't leading to memory corruption, nor will it crash the system. I would call it benign, in the context of what we're trying to do.