syzbot

map[Benign:true Confident:true Explanation:The data race occurs between `inode_set_ctime_current()` (the writer) and `generic_fillattr()` (the reader) on the `i_ctime_nsec` field of a `struct inode`.

### Analysis of the Race

1.  **Writer Side (`inode_set_ctime_current`):**
    The writer is updating the inode's change time (ctime). In the reported case, the inode is not using multigrain timestamps (`is_mgtime(inode)` is false), so it calls `inode_set_ctime_to_ts()`. This function performs plain assignments to the `i_ctime_sec` and `i_ctime_nsec` fields:
    ```c
    inode->i_ctime_sec = ts.tv_sec;
    inode->i_ctime_nsec = ts.tv_nsec;
    ```
    The writer (task 13256) is executing a `setxattr` operation, which holds the `inode->i_rwsem` mutex.

2.  **Reader Side (`generic_fillattr`):**
    The reader is fetching inode attributes for a `stat` system call. It calls `inode_get_ctime()`, which in turn calls `inode_get_ctime_nsec()`:
    ```c
    return inode->i_ctime_nsec & ~I_CTIME_QUERIED;
    ```
    The reader (task 3304) does not hold the `inode->i_rwsem` mutex, as `stat` operations are designed to be lockless and fast.

### Impact of the Race

The race is on the `i_ctime_nsec` field (4 bytes). On x86_64, a 4-byte aligned load or store is atomic, so the reader will see either the old value or the new value of the nanoseconds field.

The primary consequence of this race is a potential inconsistency between the `i_ctime_sec` and `i_ctime_nsec` fields. Because they are updated and read separately without a sequence counter or similar synchronization, a reader might see:
*   The new `sec` value and the old `nsec` value (potentially jumping forward in time by up to 1 second).
*   The old `sec` value and the new `nsec` value (potentially jumping backward in time by up to 1 second).

### Conclusion

This type of data race on timestamps is common in the Linux kernel VFS. Timestamps are considered informative metadata rather than critical control data. An occasional, transiently inconsistent timestamp returned by `stat()` is generally acceptable and does not lead to:
*   Memory corruption.
*   Kernel crashes or hangs.
*   Security vulnerabilities.

The recent introduction of multigrain timestamps (which use atomic operations on `i_ctime_nsec` to manage flags) addresses granularity issues but does not change the fundamental non-atomic nature of the `sec`/`nsec` pair for non-multigrain filesystems like the one in this report.

Given that the race is on a simple metadata field and has no impact on system stability or memory safety, it is classified as **benign**.]

map[KernelSrc:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/src/22b53f5d3c32fa7512dcfc9c7f3ae37c008c3955]

map[KernelObj:/usr/local/google/home/dvyukov/syzkaller/agent/workdir/cache/build/f088e7c04026f6805eaa66f1d1ae7669f331d79f]

map[Index:codesearch-index]

gemini-3-flash-preview

map[Benign:true Confident:true]

You are an experienced Linux kernel developer tasked with determining if the given kernel
data race is benign or not. The data race report is from KCSAN tool.
It contains 2 stack traces of the memory accesses that constitute a data race.

A "benign" data races are on a simple int/bool variable or similar field,
and the accesses are not supposed to be protected by any mutual exclusion primitives.
Common examples of such "benign" data races are accesses to various flags fields,
statistics counters, and similar. A "benign" data race does not lead to memory corruption/crash
with a conservative compiler that compiles memory accesses to primitive types
effectively as atomic.

A non-benign (or "harmful" data race) can lead to corruption/crash even with
a conservative compiler that compiles memory accesses to primitive types
effectively as atomic. A common example of a "harmful" data races is race on
a complex container (list/hashmap/etc), where accesses are supposed to be protected
by a mutual exclusion primitive.

Also consider races that happen at the same time with the given one.
If there is no synchronization in between, other memory accesses in the involved threads
race with each other if they access the same memory. For example, if both threads execute:

	some_struct->have_elements = true;
	list_add(new_node, &some_struct->list_head);

the race on some_struct->have_elements may appear benign, however it also implies there
is a race on some_struct->list_head which is not benign, since the list is not thread-safe.

Take into account that on 32-bit systems 64-bit memory accesses may be split into two accesses,
and thus even with a conservative compiler may not be fully atomic. However, such races may
still be benign depending on how writes are done, and how read data is used.

In the final reply explain why you think the given data race is benign or is harmful.

Use the provided tools to confirm any assumptions, variables/fields being accessed, etc.
In particular, don't make assumptions about the kernel source code,
use codesearch tools to read the actual source code.

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The data race report is:

==================================================================
BUG: KCSAN: data-race in generic_fillattr / inode_set_ctime_current

write to 0xffff8881052d1b20 of 4 bytes by task 13256 on cpu 1:
 inode_set_ctime_to_ts fs/inode.c:2779 [inline]
 inode_set_ctime_current+0x256/0x960 fs/inode.c:2842
 shmem_xattr_handler_set+0x1ac/0x220 mm/shmem.c:4355
 __vfs_setxattr+0x2e6/0x310 fs/xattr.c:200
 __vfs_setxattr_noperm+0xe8/0x410 fs/xattr.c:234
 __vfs_setxattr_locked+0x105/0x220 fs/xattr.c:295
 vfs_setxattr+0x12c/0x2a0 fs/xattr.c:321
 do_setxattr fs/xattr.c:636 [inline]
 filename_setxattr+0x1c4/0x410 fs/xattr.c:664
 path_setxattrat+0x2be/0x310 fs/xattr.c:708
 __do_sys_lsetxattr fs/xattr.c:749 [inline]
 __se_sys_lsetxattr fs/xattr.c:745 [inline]
 __x64_sys_lsetxattr+0x71/0x90 fs/xattr.c:745
 x64_sys_call+0x1300/0x3020 arch/x86/include/generated/asm/syscalls_64.h:190
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x370 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

read to 0xffff8881052d1b20 of 4 bytes by task 3304 on cpu 0:
 inode_get_ctime_nsec include/linux/fs.h:1689 [inline]
 inode_get_ctime include/linux/fs.h:1695 [inline]
 generic_fillattr+0x1d1/0x340 fs/stat.c:101
 shmem_getattr+0x181/0x200 mm/shmem.c:1302
 vfs_getattr_nosec+0x146/0x1e0 fs/stat.c:213
 vfs_getattr+0x48/0x60 fs/stat.c:262
 vfs_statx_path+0x28/0x140 fs/stat.c:299
 vfs_statx+0xc4/0x170 fs/stat.c:356
 vfs_fstatat+0x118/0x170 fs/stat.c:373
 __do_sys_newfstatat fs/stat.c:538 [inline]
 __se_sys_newfstatat+0x55/0x3e0 fs/stat.c:532
 __x64_sys_newfstatat+0x55/0x70 fs/stat.c:532
 x64_sys_call+0x2c64/0x3020 arch/x86/include/generated/asm/syscalls_64.h:263
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x370 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

value changed: 0x318f95a9 -> 0x0efd7cb1

Reported by Kernel Concurrency Sanitizer on:
CPU: 0 UID: 0 PID: 3304 Comm: udevd Tainted: G        W           syzkaller #0 PREEMPT(full) 
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 02/12/2026
==================================================================

The data race occurs between `inode_set_ctime_current()` (the writer) and `generic_fillattr()` (the reader) on the `i_ctime_nsec` field of a `struct inode`.

### Analysis of the Race

1.  **Writer Side (`inode_set_ctime_current`):**
    The writer is updating the inode's change time (ctime). In the reported case, the inode is not using multigrain timestamps (`is_mgtime(inode)` is false), so it calls `inode_set_ctime_to_ts()`. This function performs plain assignments to the `i_ctime_sec` and `i_ctime_nsec` fields:
    ```c
    inode->i_ctime_sec = ts.tv_sec;
    inode->i_ctime_nsec = ts.tv_nsec;
    ```
    The writer (task 13256) is executing a `setxattr` operation, which holds the `inode->i_rwsem` mutex.

2.  **Reader Side (`generic_fillattr`):**
    The reader is fetching inode attributes for a `stat` system call. It calls `inode_get_ctime()`, which in turn calls `inode_get_ctime_nsec()`:
    ```c
    return inode->i_ctime_nsec & ~I_CTIME_QUERIED;
    ```
    The reader (task 3304) does not hold the `inode->i_rwsem` mutex, as `stat` operations are designed to be lockless and fast.

### Impact of the Race

The race is on the `i_ctime_nsec` field (4 bytes). On x86_64, a 4-byte aligned load or store is atomic, so the reader will see either the old value or the new value of the nanoseconds field.

The primary consequence of this race is a potential inconsistency between the `i_ctime_sec` and `i_ctime_nsec` fields. Because they are updated and read separately without a sequence counter or similar synchronization, a reader might see:
*   The new `sec` value and the old `nsec` value (potentially jumping forward in time by up to 1 second).
*   The old `sec` value and the new `nsec` value (potentially jumping backward in time by up to 1 second).

### Conclusion

This type of data race on timestamps is common in the Linux kernel VFS. Timestamps are considered informative metadata rather than critical control data. An occasional, transiently inconsistent timestamp returned by `stat()` is generally acceptable and does not lead to:
*   Memory corruption.
*   Kernel crashes or hangs.
*   Security vulnerabilities.

The recent introduction of multigrain timestamps (which use atomic operations on `i_ctime_nsec` to manage flags) addresses granularity issues but does not change the fundamental non-atomic nature of the `sec`/`nsec` pair for non-multigrain filesystems like the one in this report.

Given that the race is on a simple metadata field and has no impact on system stability or memory safety, it is classified as **benign**.

gemini-3-flash-preview

							input: 4218
							output: 122
							thoughts: 147
						

**Analyzing a Data Race on Inode ctime**

Okay, so the report flags a nasty data race – *again* – this time between `inode_set_ctime_current` and `generic_fillattr`, both operating on an inode's ctime. We're talking a 4-byte write, followed by a 4-byte read, occurring at that specific memory address: `0xffff8881052d1b20`. That's the smoking gun, or at least the starting point.

The write, according to the report, stems from within `inode_set_ctime_to_ts`, which is neatly inlined into `inode_set_ctime_current`. Understandable. The read, on the other hand, comes through a chain: `inode_get_ctime_nsec`, which is inlined into `inode_get_ctime`, and *that's* inlined into `generic_fillattr`. That tells me we're likely dealing with something relatively fundamental.

Time to dive into the code for those functions, both to understand *how* they're accessing ctime and, more importantly, *why* they're not coordinating properly. I need to figure out what's missing in the synchronization here; a proper lock, an atomic operation, something. Let's see what we're working with. This could be anything from a simple missed lock, to an improper use of read-copy-update (RCU), or even some subtle ordering issue. I need to figure out what the root cause is, because fixing *this* kind of bug requires precision and a good understanding of the system's architecture.

map[ContextFile:fs/inode.c IncludeLines:true Name:inode_set_ctime_current]

map[SourceCode:2815:	/**
2816:	 * inode_set_ctime_current - set the ctime to current_time
2817:	 * @inode: inode
2818:	 *
2819:	 * Set the inode's ctime to the current value for the inode. Returns the
2820:	 * current value that was assigned. If this is not a multigrain inode, then we
2821:	 * set it to the later of the coarse time and floor value.
2822:	 *
2823:	 * If it is multigrain, then we first see if the coarse-grained timestamp is
2824:	 * distinct from what is already there. If so, then use that. Otherwise, get a
2825:	 * fine-grained timestamp.
2826:	 *
2827:	 * After that, try to swap the new value into i_ctime_nsec. Accept the
2828:	 * resulting ctime, regardless of the outcome of the swap. If it has
2829:	 * already been replaced, then that timestamp is later than the earlier
2830:	 * unacceptable one, and is thus acceptable.
2831:	 */
2832:	struct timespec64 inode_set_ctime_current(struct inode *inode)
2833:	{
2834:		struct timespec64 now;
2835:		u32 cns, cur;
2836:	
2837:		ktime_get_coarse_real_ts64_mg(&now);
2838:		now = timestamp_truncate(now, inode);
2839:	
2840:		/* Just return that if this is not a multigrain fs */
2841:		if (!is_mgtime(inode)) {
2842:			inode_set_ctime_to_ts(inode, now);
2843:			goto out;
2844:		}
2845:	
2846:		/*
2847:		 * A fine-grained time is only needed if someone has queried
2848:		 * for timestamps, and the current coarse grained time isn't
2849:		 * later than what's already there.
2850:		 */
2851:		cns = smp_load_acquire(&inode->i_ctime_nsec);
2852:		if (cns & I_CTIME_QUERIED) {
2853:			struct timespec64 ctime = { .tv_sec = inode->i_ctime_sec,
2854:						    .tv_nsec = cns & ~I_CTIME_QUERIED };
2855:	
2856:			if (timespec64_compare(&now, &ctime) <= 0) {
2857:				ktime_get_real_ts64_mg(&now);
2858:				now = timestamp_truncate(now, inode);
2859:				mgtime_counter_inc(mg_fine_stamps);
2860:			}
2861:		}
2862:		mgtime_counter_inc(mg_ctime_updates);
2863:	
2864:		/* No need to cmpxchg if it's exactly the same */
2865:		if (cns == now.tv_nsec && inode->i_ctime_sec == now.tv_sec) {
2866:			trace_ctime_xchg_skip(inode, &now);
2867:			goto out;
2868:		}
2869:		cur = cns;
2870:	retry:
2871:		/* Try to swap the nsec value into place. */
2872:		if (try_cmpxchg(&inode->i_ctime_nsec, &cur, now.tv_nsec)) {
2873:			/* If swap occurred, then we're (mostly) done */
2874:			inode->i_ctime_sec = now.tv_sec;
2875:			trace_ctime_ns_xchg(inode, cns, now.tv_nsec, cur);
2876:			mgtime_counter_inc(mg_ctime_swaps);
2877:		} else {
2878:			/*
2879:			 * Was the change due to someone marking the old ctime QUERIED?
2880:			 * If so then retry the swap. This can only happen once since
2881:			 * the only way to clear I_CTIME_QUERIED is to stamp the inode
2882:			 * with a new ctime.
2883:			 */
2884:			if (!(cns & I_CTIME_QUERIED) && (cns | I_CTIME_QUERIED) == cur) {
2885:				cns = cur;
2886:				goto retry;
2887:			}
2888:			/* Otherwise, keep the existing ctime */
2889:			now.tv_sec = inode->i_ctime_sec;
2890:			now.tv_nsec = cur & ~I_CTIME_QUERIED;
2891:		}
2892:	out:
2893:		return now;
2894:	}
 SourceFile:fs/inode.c]

map[ContextFile:fs/stat.c IncludeLines:true Name:generic_fillattr]

map[SourceCode:  65:	/**
  66:	 * generic_fillattr - Fill in the basic attributes from the inode struct
  67:	 * @idmap:		idmap of the mount the inode was found from
  68:	 * @request_mask:	statx request_mask
  69:	 * @inode:		Inode to use as the source
  70:	 * @stat:		Where to fill in the attributes
  71:	 *
  72:	 * Fill in the basic attributes in the kstat structure from data that's to be
  73:	 * found on the VFS inode structure.  This is the default if no getattr inode
  74:	 * operation is supplied.
  75:	 *
  76:	 * If the inode has been found through an idmapped mount the idmap of
  77:	 * the vfsmount must be passed through @idmap. This function will then
  78:	 * take care to map the inode according to @idmap before filling in the
  79:	 * uid and gid filds. On non-idmapped mounts or if permission checking is to be
  80:	 * performed on the raw inode simply pass @nop_mnt_idmap.
  81:	 */
  82:	void generic_fillattr(struct mnt_idmap *idmap, u32 request_mask,
  83:			      struct inode *inode, struct kstat *stat)
  84:	{
  85:		vfsuid_t vfsuid = i_uid_into_vfsuid(idmap, inode);
  86:		vfsgid_t vfsgid = i_gid_into_vfsgid(idmap, inode);
  87:	
  88:		stat->dev = inode->i_sb->s_dev;
  89:		stat->ino = inode->i_ino;
  90:		stat->mode = inode->i_mode;
  91:		stat->nlink = inode->i_nlink;
  92:		stat->uid = vfsuid_into_kuid(vfsuid);
  93:		stat->gid = vfsgid_into_kgid(vfsgid);
  94:		stat->rdev = inode->i_rdev;
  95:		stat->size = i_size_read(inode);
  96:		stat->atime = inode_get_atime(inode);
  97:	
  98:		if (is_mgtime(inode)) {
  99:			fill_mg_cmtime(stat, request_mask, inode);
 100:		} else {
 101:			stat->ctime = inode_get_ctime(inode);
 102:			stat->mtime = inode_get_mtime(inode);
 103:		}
 104:	
 105:		stat->blksize = i_blocksize(inode);
 106:		stat->blocks = inode->i_blocks;
 107:	
 108:		if ((request_mask & STATX_CHANGE_COOKIE) && IS_I_VERSION(inode)) {
 109:			stat->result_mask |= STATX_CHANGE_COOKIE;
 110:			stat->change_cookie = inode_query_iversion(inode);
 111:		}
 112:	
 113:	}
 SourceFile:fs/stat.c]

map[ContextFile:include/linux/fs.h IncludeLines:true Name:inode_get_ctime_nsec]

map[SourceCode:1687:	static inline long inode_get_ctime_nsec(const struct inode *inode)
1688:	{
1689:		return inode->i_ctime_nsec & ~I_CTIME_QUERIED;
1690:	}
 SourceFile:include/linux/fs.h]

gemini-3-flash-preview

							input: 7170
							output: 84
							thoughts: 401
						

**Analyzing the `inode->i_ctime_nsec` Race Condition**

Okay, so the issue boils down to a race condition around `inode->i_ctime_nsec`. The trace indicates a write to this memory location by task 13256 in `inode_set_ctime_to_ts`, which is inlined within `inode_set_ctime_current`. Then, a read from the same location is performed by task 3304 in `inode_get_ctime_nsec`, which is itself inlined inside of `generic_fillattr`.

Essentially, if an inode is *not* multigrain, `inode_set_ctime_current` calls `inode_set_ctime_to_ts`. Then, `generic_fillattr` calls `inode_get_ctime`. In the code, `inode_set_ctime_to_ts` writes the value to `i_ctime_nsec`, and `inode_get_ctime_nsec` reads that same value.

The code snippets support this: `inode_set_ctime_to_ts` writes to `i_ctime_nsec` when `!is_mgtime(inode)` and `inode_get_ctime_nsec` reads from `i_ctime_nsec` when `!is_mgtime(inode)`. I need to dig deeper into those two functions now to understand the synchronization (or lack thereof) involved and the exact nature of the race.

map[ContextFile:fs/inode.c IncludeLines:true Name:inode_set_ctime_to_ts]

map[SourceCode:2774:	struct timespec64 inode_set_ctime_to_ts(struct inode *inode, struct timespec64 ts)
2775:	{
2776:		trace_inode_set_ctime_to_ts(inode, &ts);
2777:		set_normalized_timespec64(&ts, ts.tv_sec, ts.tv_nsec);
2778:		inode->i_ctime_sec = ts.tv_sec;
2779:		inode->i_ctime_nsec = ts.tv_nsec;
2780:		return ts;
2781:	}
 SourceFile:fs/inode.c]

map[ContextFile:include/linux/fs.h IncludeLines:true Name:inode_get_ctime]

map[SourceCode:1692:	static inline struct timespec64 inode_get_ctime(const struct inode *inode)
1693:	{
1694:		struct timespec64 ts = { .tv_sec  = inode_get_ctime_sec(inode),
1695:					 .tv_nsec = inode_get_ctime_nsec(inode) };
1696:	
1697:		return ts;
1698:	}
 SourceFile:include/linux/fs.h]

gemini-3-flash-preview

							input: 8287
							output: 44
							thoughts: 175
						

**Analyzing a Potential Race Condition in the SHMEM Filesystem**

Okay, so I'm looking at this code and immediately see a potential race condition involving `i_ctime_nsec`. On one side, we have a writer updating the inode's `i_ctime_nsec` field with code that looks like this:

```c
inode->i_ctime_sec = ts.tv_sec;
inode->i_ctime_nsec = ts.tv_nsec;
```

And on the other side, a reader is grabbing those values:

```c
struct timespec64 ts = { .tv_sec  = inode_get_ctime_sec(inode),
					 .tv_nsec = inode_get_ctime_nsec(inode) };
```

My immediate concern is whether any locking mechanisms are in place to synchronize access to `i_ctime_nsec`. This looks like a classic read-modify-write scenario that's ripe for trouble if not handled carefully. The writer, specifically, is identified as `shmem_xattr_handler_set`, and the reader as `shmem_getattr`. The next step is to drill into `shmem_xattr_handler_set` and see what's happening. I need to understand the execution context and any potential concurrency controls that are being used here. If there's no protection, then we have a classic data race, and depending on how this value is used later, we could have file metadata consistency issues.

map[ContextFile:mm/shmem.c IncludeLines:true Name:shmem_xattr_handler_set]

map[SourceCode:4323:	static int shmem_xattr_handler_set(const struct xattr_handler *handler,
4324:					   struct mnt_idmap *idmap,
4325:					   struct dentry *unused, struct inode *inode,
4326:					   const char *name, const void *value,
4327:					   size_t size, int flags)
4328:	{
4329:		struct shmem_inode_info *info = SHMEM_I(inode);
4330:		struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
4331:		struct simple_xattr *old_xattr;
4332:		size_t ispace = 0;
4333:	
4334:		name = xattr_full_name(handler, name);
4335:		if (value && sbinfo->max_inodes) {
4336:			ispace = simple_xattr_space(name, size);
4337:			raw_spin_lock(&sbinfo->stat_lock);
4338:			if (sbinfo->free_ispace < ispace)
4339:				ispace = 0;
4340:			else
4341:				sbinfo->free_ispace -= ispace;
4342:			raw_spin_unlock(&sbinfo->stat_lock);
4343:			if (!ispace)
4344:				return -ENOSPC;
4345:		}
4346:	
4347:		old_xattr = simple_xattr_set(&info->xattrs, name, value, size, flags);
4348:		if (!IS_ERR(old_xattr)) {
4349:			ispace = 0;
4350:			if (old_xattr && sbinfo->max_inodes)
4351:				ispace = simple_xattr_space(old_xattr->name,
4352:							    old_xattr->size);
4353:			simple_xattr_free(old_xattr);
4354:			old_xattr = NULL;
4355:			inode_set_ctime_current(inode);
4356:			inode_inc_iversion(inode);
4357:		}
4358:		if (ispace) {
4359:			raw_spin_lock(&sbinfo->stat_lock);
4360:			sbinfo->free_ispace += ispace;
4361:			raw_spin_unlock(&sbinfo->stat_lock);
4362:		}
4363:		return PTR_ERR(old_xattr);
4364:	}
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 9552
							output: 39
							thoughts: 90
						

**Analyzing the Potential Race Condition**

Okay, so I'm looking at this `shmem_xattr_handler_set` function, and it's called from `vfs_setxattr`. My initial thought is that this should be safe since `vfs_setxattr` typically acquires the `inode->i_rwsem` semaphore in write mode. That generally provides the necessary protection, but I need to make sure.

Now, I'm also seeing that `shmem_getattr` is called from `vfs_getattr`. That's a different story. `vfs_getattr` *usually* doesn't require holding `inode->i_rwsem` in write mode, but maybe read mode? Or possibly not at all? It's been a while since I looked that far into the architecture. I really need to verify if this is holding the inode lock or not.

The key question here is whether or not there's potential for a race condition between a write operation through `shmem_xattr_handler_set` and a read operation through `shmem_getattr`. I need to dig into the `shmem_getattr` implementation to understand exactly how it interacts with the shared memory inode. It's time to check out the details of `shmem_getattr`.

map[ContextFile:mm/shmem.c IncludeLines:true Name:shmem_getattr]

map[SourceCode:1283:	static int shmem_getattr(struct mnt_idmap *idmap,
1284:				 const struct path *path, struct kstat *stat,
1285:				 u32 request_mask, unsigned int query_flags)
1286:	{
1287:		struct inode *inode = path->dentry->d_inode;
1288:		struct shmem_inode_info *info = SHMEM_I(inode);
1289:	
1290:		if (info->alloced - info->swapped != inode->i_mapping->nrpages)
1291:			shmem_recalc_inode(inode, 0, 0);
1292:	
1293:		if (info->fsflags & FS_APPEND_FL)
1294:			stat->attributes |= STATX_ATTR_APPEND;
1295:		if (info->fsflags & FS_IMMUTABLE_FL)
1296:			stat->attributes |= STATX_ATTR_IMMUTABLE;
1297:		if (info->fsflags & FS_NODUMP_FL)
1298:			stat->attributes |= STATX_ATTR_NODUMP;
1299:		stat->attributes_mask |= (STATX_ATTR_APPEND |
1300:				STATX_ATTR_IMMUTABLE |
1301:				STATX_ATTR_NODUMP);
1302:		generic_fillattr(idmap, request_mask, inode, stat);
1303:	
1304:		if (shmem_huge_global_enabled(inode, 0, 0, false, NULL, 0))
1305:			stat->blksize = HPAGE_PMD_SIZE;
1306:	
1307:		if (request_mask & STATX_BTIME) {
1308:			stat->result_mask |= STATX_BTIME;
1309:			stat->btime.tv_sec = info->i_crtime.tv_sec;
1310:			stat->btime.tv_nsec = info->i_crtime.tv_nsec;
1311:		}
1312:	
1313:		return 0;
1314:	}
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 10556
							output: 41
							thoughts: 786
						

**Analyzing the Ctime Race Condition in Shared Memory**

Okay, so I'm looking at this `shmem_getattr` function, which calls `generic_fillattr`, and I'm thinking about the potential race condition when updating and reading the ctime timestamp. It's just a timestamp, so the consequences of reading an old or new value seem relatively benign, right? But then I remember that `timespec64` is composed of two fields: `tv_sec` and `tv_nsec`. That's where things get interesting.

If these fields are updated separately, without any sort of locking, a reader could end up seeing an inconsistent value – a mix of old seconds and new nanoseconds, or vice versa. The code snippet for updating `ctime` is pretty straightforward:

```c
inode->i_ctime_sec = ts.tv_sec;
inode->i_ctime_nsec = ts.tv_nsec;
```

And reading is just as simple:

```c
struct timespec64 ts = { .tv_sec  = inode_get_ctime_sec(inode),
                         .tv_nsec = inode_get_ctime_nsec(inode) };
```

Basically, these functions just read the individual fields.

So, let's say `ctime` is transitioning from 10.999 seconds to 11.001 seconds. Here's a possible scenario: the writer updates `i_ctime_sec` to 11, then updates `i_ctime_nsec` to 1000000. Meanwhile, the reader reads `i_ctime_sec` as 11, and then reads `i_ctime_nsec` as 999000000 (the old value). The result? An inconsistent 11.999 seconds, which is actually *in the future* compared to both the old and new values. That's a definite problem.

But wait, I see `inode_set_ctime_current` has some logic related to multigrain inodes. This is interesting.

```c
if (!is_mgtime(inode)) {
  inode_set_ctime_to_ts(inode, now);
  goto out;
}
```

If it *isn't* multigrain, it just uses the simple two-step update, which is vulnerable to the race condition. If it *is* multigrain, it uses `try_cmpxchg` on `i_ctime_nsec`. But… it still updates `i_ctime_sec` separately:

```c
if (try_cmpxchg(&inode->i_ctime_nsec, &cur, now.tv_nsec)) {
  inode->i_ctime_sec = now.tv_sec;
```

So, for multigrain inodes, the logic first updates `i_ctime_nsec` with a compare-and-exchange operation, and then updates `i_ctime_sec`. But for *non* multigrain inodes, it's updating the `i_ctime_sec` and then `i_ctime_nsec` fields. In both scenarios, there isn't a single, atomic update of both fields.

The crucial question is, is `shmem` multigrain? I need to check `shmem_init_inode` to determine this.

map[ContextFile:mm/shmem.c IncludeLines:true Name:shmem_get_inode]

map[SourceCode:3175:	static struct inode *shmem_get_inode(struct mnt_idmap *idmap,
3176:					     struct super_block *sb, struct inode *dir,
3177:					     umode_t mode, dev_t dev, vma_flags_t flags)
3178:	{
3179:		return __shmem_get_inode(idmap, sb, dir, mode, dev, flags);
3180:	}
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 12255
							output: 42
							thoughts: 13
						

Okay, here's my summarized thought process, presented as if I were considering the implications myself, and assuming the user is a knowledgeable expert:

**Investigating `__shmem_get_inode`**

Alright, time to dive into this. Let's start by taking a close look at `__shmem_get_inode`. Understanding how this specific function operates is crucial for grasping the broader memory management strategy within this shared memory system. I need to figure out exactly what its role is in creating, retrieving, or manipulating inodes within the shared memory region. Is this a core building block, a helper, or perhaps something more specialized? I need to understand its relationship to the other inode-related functions, and what other components of the kernel rely on its output. From a security perspective, I need to know how it handles potential race conditions, or access control. Knowing the code intimately will allow me to assess the impact of any changes or identify areas for potential optimization. Where is this function called from? What are the calling contexts, and what preconditions are established before it's invoked? Does it implement any synchronization primitives? I also need to think about error handling - what happens in exceptional situations? How are problems detected, and what kind of error codes are returned? Finally, what are the potential performance implications of this function? Does it introduce any bottlenecks, or create any additional overhead, when called frequently? Let's get to the bottom of this.

map[ContextFile:mm/shmem.c IncludeLines:true Name:__shmem_get_inode]

map[SourceCode:3064:	static struct inode *__shmem_get_inode(struct mnt_idmap *idmap,
3065:					       struct super_block *sb,
3066:					       struct inode *dir, umode_t mode,
3067:					       dev_t dev, vma_flags_t flags)
3068:	{
3069:		struct inode *inode;
3070:		struct shmem_inode_info *info;
3071:		struct shmem_sb_info *sbinfo = SHMEM_SB(sb);
3072:		ino_t ino;
3073:		int err;
3074:	
3075:		err = shmem_reserve_inode(sb, &ino);
3076:		if (err)
3077:			return ERR_PTR(err);
3078:	
3079:		inode = new_inode(sb);
3080:		if (!inode) {
3081:			shmem_free_inode(sb, 0);
3082:			return ERR_PTR(-ENOSPC);
3083:		}
3084:	
3085:		inode->i_ino = ino;
3086:		inode_init_owner(idmap, inode, dir, mode);
3087:		inode->i_blocks = 0;
3088:		simple_inode_init_ts(inode);
3089:		inode->i_generation = get_random_u32();
3090:		info = SHMEM_I(inode);
3091:		memset(info, 0, (char *)inode - (char *)info);
3092:		spin_lock_init(&info->lock);
3093:		atomic_set(&info->stop_eviction, 0);
3094:		info->seals = F_SEAL_SEAL;
3095:		info->flags = vma_flags_test(&flags, VMA_NORESERVE_BIT)
3096:			? SHMEM_F_NORESERVE : 0;
3097:		info->i_crtime = inode_get_mtime(inode);
3098:		info->fsflags = (dir == NULL) ? 0 :
3099:			SHMEM_I(dir)->fsflags & SHMEM_FL_INHERITED;
3100:		if (info->fsflags)
3101:			shmem_set_inode_flags(inode, info->fsflags, NULL);
3102:		INIT_LIST_HEAD(&info->shrinklist);
3103:		INIT_LIST_HEAD(&info->swaplist);
3104:		simple_xattrs_init(&info->xattrs);
3105:		cache_no_acl(inode);
3106:		if (sbinfo->noswap)
3107:			mapping_set_unevictable(inode->i_mapping);
3108:	
3109:		/* Don't consider 'deny' for emergencies and 'force' for testing */
3110:		if (sbinfo->huge)
3111:			mapping_set_large_folios(inode->i_mapping);
3112:	
3113:		switch (mode & S_IFMT) {
3114:		default:
3115:			inode->i_op = &shmem_special_inode_operations;
3116:			init_special_inode(inode, mode, dev);
3117:			break;
3118:		case S_IFREG:
3119:			inode->i_mapping->a_ops = &shmem_aops;
3120:			inode->i_op = &shmem_inode_operations;
3121:			inode->i_fop = &shmem_file_operations;
3122:			mpol_shared_policy_init(&info->policy,
3123:						 shmem_get_sbmpol(sbinfo));
3124:			break;
3125:		case S_IFDIR:
3126:			inc_nlink(inode);
3127:			/* Some things misbehave if size == 0 on a directory */
3128:			inode->i_size = 2 * BOGO_DIRENT_SIZE;
3129:			inode->i_op = &shmem_dir_inode_operations;
3130:			inode->i_fop = &simple_offset_dir_operations;
3131:			simple_offset_init(shmem_get_offset_ctx(inode));
3132:			break;
3133:		case S_IFLNK:
3134:			/*
3135:			 * Must not load anything in the rbtree,
3136:			 * mpol_free_shared_policy will not be called.
3137:			 */
3138:			mpol_shared_policy_init(&info->policy, NULL);
3139:			break;
3140:		}
3141:	
3142:		lockdep_annotate_inode_mutex_key(inode);
3143:		return inode;
3144:	}
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 13977
							output: 41
							thoughts: 30
						

Okay, here's my interpretation of those thoughts, summarized from my perspective as an expert in the field:

**Debugging a Missing Filesystem Flag**

Hmm, I'm not observing the `FS_MGTIME` flag being set for shared memory. That's peculiar. It's often critical for ensuring proper metadata updates, and its absence could indicate a subtle bug. Time to investigate the root cause. My first instinct is to dive into how the shared memory filesystem type, specifically `shmem_fs_type`, is being handled. I need to understand the control flow that governs whether `FS_MGTIME` is being set and, if not, why not. Is there a conditional that's failing, a bit that's being masked out, or is it simply not being applied at all? Time to dig deeper. I'm going to start tracing through the code and examining related data structures to find where this is going wrong.

map[ContextFile:mm/shmem.c IncludeLines:true Name:shmem_fs_type]

map[SourceCode:5351:	static struct file_system_type shmem_fs_type = {
5352:		.owner		= THIS_MODULE,
5353:		.name		= "tmpfs",
5354:		.init_fs_context = shmem_init_fs_context,
5355:	#ifdef CONFIG_TMPFS
5356:		.parameters	= shmem_fs_parameters,
5357:	#endif
5358:		.kill_sb	= kill_anon_super,
5359:		.fs_flags	= FS_USERNS_MOUNT | FS_ALLOW_IDMAP | FS_MGTIME,
5360:	};
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 14448
							output: 40
							thoughts: 67
						

Okay, let's break this down.

**Understanding Memory Granularity Time (MGTIME) in Filesystem Operations**

Right, so I'm looking at this code and I see `FS_MGTIME` listed in `fs_flags`. That immediately tells me something. That means the system should be tracking memory granularity time (MGTIME) for this filesystem. Specifically, it should mean that the function `is_mgtime(inode)` is returning true for shared memory (shmem) inodes. That's the logical connection I'm making.

Now I'm pausing for a second. Hang on, I need to confirm something. I need to make *absolutely* sure what that function, `is_mgtime`, is checking. Is it looking at the filesystem flags (`s_flags`) or the inode flags (`i_flags`)? It's a key distinction and critical for understanding how the time tracking is handled. I need to refresh my memory on the specific implementation of that check. Okay, time to go look at `is_mgtime` itself to clarify which flags are being used.

map[ContextFile:include/linux/fs.h IncludeLines:true Name:is_mgtime]

map[SourceCode:2304:	/**
2305:	 * is_mgtime: is this inode using multigrain timestamps
2306:	 * @inode: inode to test for multigrain timestamps
2307:	 *
2308:	 * Return true if the inode uses multigrain timestamps, false otherwise.
2309:	 */
2310:	static inline bool is_mgtime(const struct inode *inode)
2311:	{
2312:		return inode->i_opflags & IOP_MGTIME;
2313:	}
 SourceFile:include/linux/fs.h]

gemini-3-flash-preview

							input: 14956
							output: 23
							thoughts: 43
						

**Tracing the Source of `IOP_MGTIME`**

Okay, so I'm trying to figure out where the `IOP_MGTIME` flag is getting set. My initial instinct is that it's likely initialized in the core filesystem structures. Specifically, I'm thinking about the `inode_init_always` function, or potentially during superblock initialization. That's where you'd typically expect to find flags like this being set for inodes.

Right, so to get a handle on the situation I need to trace the references. I'll search the code base for all occurrences of `IOP_MGTIME`. That should give me the answer quickly.

requested entity does not exist

map[Name:IOP_MGTIME]

map[References:<nil> TruncatedOutput:false]

gemini-3-flash-preview

							input: 15184
							output: 19
							thoughts: 13
						

Okay, here's the summary, assuming I'm a seasoned professional, and the "it" refers to some specific element in my field:

**Internal Investigation - Tracking Down the Assignment**

Alright, I need to backtrack. I've clearly jumped the gun or lost my place. My initial approach was off-base. Rather than diving into the specifics just yet, I need to pause and find the root. My train of thought needs a reset. I immediately should be searching for where *it* is originally assigned or defined within the system, project, or relevant documentation. This is standard procedure. Knowing where *it* originates is crucial for understanding its purpose and its current state. I should have started there. Jumping in without this foundational understanding risks wasted effort and potential misunderstandings down the line. Let me get my bearings and go back to the source.

map[Expression:IOP_MGTIME]

map[Output:fs/inode.c=228=int inode_init_always_gfp(struct super_block *sb, struct inode *inode, gfp_t gfp)
--
fs/inode.c-247-	if (sb->s_type->fs_flags & FS_MGTIME)
fs/inode.c:248:		inode->i_opflags |= IOP_MGTIME;
fs/inode.c-249-	i_uid_write(inode, 0);
--
include/linux/fs.h=621=is_uncached_acl(struct posix_acl *acl)
--
include/linux/fs.h-630-#define IOP_DEFAULT_READLINK	0x0010
include/linux/fs.h:631:#define IOP_MGTIME		0x0020
include/linux/fs.h-632-#define IOP_CACHED_LINK		0x0040
--
include/linux/fs.h=2310=static inline bool is_mgtime(const struct inode *inode)
include/linux/fs.h-2311-{
include/linux/fs.h:2312:	return inode->i_opflags & IOP_MGTIME;
include/linux/fs.h-2313-}
]

gemini-3-flash-preview

							input: 15721
							output: 40
							thoughts: 921
						

**Analyzing a Time Modification Issue in a Shmem Filesystem**

Okay, let's break this down. The report indicates a read of a ctime value within `generic_fillattr`, but the trace points to an issue with `inode_set_ctime_current`. My initial thought was: shmem inodes should have `FS_MGTIME` set through `shmem_fs_type`, causing `is_mgtime(inode)` to return true, and triggering the `fill_mg_cmtime` path. That part is clear, and the code confirms it.

The interesting part is that the reported issue is in the *else* path of `generic_fillattr`, specifically line 101, which calls `inode_get_ctime(inode)`. This means `is_mgtime(inode)` is false.  But the shmem write-path, specifically in `shmem_xattr_handler_set` calls `inode_set_ctime_current`. This sets the ctime which, based on the `is_mgtime` check in `inode_set_ctime_current` *should* use atomic updates, but it doesn't seem to be.

The stacktrace for the write, however, indicates a call to `inode_set_ctime_to_ts`, which is inside `inode_set_ctime_current` and should be bypassed if `is_mgtime(inode)` is true. That function *should* only be called if `is_mgtime(inode)` returns false. The write accesses the ctime directly, which shouldn't happen for a `FS_MGTIME` filesystem.

So, the critical point is that the *reader* and the *writer* are seemingly seeing different states for `is_mgtime(inode)`. This is a very perplexing issue. The reader is taking the non-mgtime path, the writer is taking the non-mgtime path.

The `syzkaller #0 PREEMPT(full)` might indicate we are looking at something in a different version of the kernel.

Now, let's refocus on the `fill_mg_cmtime` function itself to see if the issue is *inside* that function.

map[ContextFile:fs/stat.c IncludeLines:true Name:fill_mg_cmtime]

map[SourceCode:  31:	/**
  32:	 * fill_mg_cmtime - Fill in the mtime and ctime and flag ctime as QUERIED
  33:	 * @stat: where to store the resulting values
  34:	 * @request_mask: STATX_* values requested
  35:	 * @inode: inode from which to grab the c/mtime
  36:	 *
  37:	 * Given @inode, grab the ctime and mtime out if it and store the result
  38:	 * in @stat. When fetching the value, flag it as QUERIED (if not already)
  39:	 * so the next write will record a distinct timestamp.
  40:	 *
  41:	 * NB: The QUERIED flag is tracked in the ctime, but we set it there even
  42:	 * if only the mtime was requested, as that ensures that the next mtime
  43:	 * change will be distinct.
  44:	 */
  45:	void fill_mg_cmtime(struct kstat *stat, u32 request_mask, struct inode *inode)
  46:	{
  47:		atomic_t *pcn = (atomic_t *)&inode->i_ctime_nsec;
  48:	
  49:		/* If neither time was requested, then don't report them */
  50:		if (!(request_mask & (STATX_CTIME|STATX_MTIME))) {
  51:			stat->result_mask &= ~(STATX_CTIME|STATX_MTIME);
  52:			return;
  53:		}
  54:	
  55:		stat->mtime = inode_get_mtime(inode);
  56:		stat->ctime.tv_sec = inode->i_ctime_sec;
  57:		stat->ctime.tv_nsec = (u32)atomic_read(pcn);
  58:		if (!(stat->ctime.tv_nsec & I_CTIME_QUERIED))
  59:			stat->ctime.tv_nsec = ((u32)atomic_fetch_or(I_CTIME_QUERIED, pcn));
  60:		stat->ctime.tv_nsec &= ~I_CTIME_QUERIED;
  61:		trace_fill_mg_cmtime(inode, &stat->ctime, &stat->mtime);
  62:	}
 SourceFile:fs/stat.c]

gemini-3-flash-preview

							input: 17771
							output: 32
							thoughts: 1382
						

**Analyzing a Data Race on Inode Timestamps**

Okay, so KCSAN has flagged a potential data race, and my expert intuition is tingling. Let's break this down systematically.

First, I see a situation where `is_mgtime(inode)` *should* be true if the multigrain timestamp mechanism is active. If it *were* true, the code would be using atomic operations on `i_ctime_nsec`, which would avoid the data race, even with the slightly questionable cast to `atomic_t`. But the critical point is: the report says that the problem is *not* in `fill_mg_cmtime`.

Instead, the report points to a read in `inode_get_ctime_nsec` and a write in `inode_set_ctime_to_ts`. Crucially, these functions are used when `is_mgtime(inode)` is FALSE. I had a quick moment of doubt, thinking that maybe `shmem` was using multigrain timestamps after all, but after double-checking the code path with a bit more of a stack trace... Nope, `is_mgtime(inode)` is definitely false. So this shmem object is not using the multigrain timestamps in this kernel version.

So, we have a plain data race on `i_ctime_sec` and `i_ctime_nsec`. The race is on the *raw* values. The report only mentions `i_ctime_nsec`, but `i_ctime_sec` is also likely racing, and I have to remember that they are adjacent!

Is this harmful? Well, if the writer and reader don't synchronize, a reader can see an inconsistent timestamp.  A timestamp from the future is *worse* than one from the past, as it can cause problems in applications that check for relative ordering.  While "jumping backward" might be "fine" in some interpretations of filesystem timestamps, seeing something in the future, if it can occur (and it can!), will definitely break applications.

However, a critical point comes up: is this the "standard" race for timestamps? The writer seems to hold `inode_lock(inode)` within `shmem_xattr_handler_set`, which is called through `vfs_setxattr` and eventually, through the locking mechanisms inherent in xattr setting. But does the *reader* hold the lock? `vfs_getattr` does *not* hold `inode_lock(inode)`. This is common; `stat()` is usually optimized for speed, and we don't want to block on a mutex.

Okay, so we have a writer holding a lock, and a reader holding *nothing*. Classic data race.

Is it benign? Many filesystems update timestamps like this. But if it leads to inconsistent timestamps, it might be considered a bug.

Let's look at the data types: `i_ctime_sec` is 64-bit (`time64_t`), `i_ctime_nsec` is 32-bit (`unsigned int`). The update is atomic and the read is atomic, but they are *separate* reads and writes!

If the reader sees a new `sec` but the old `nsec`, the timestamp jumps forward. If it sees the old `sec` but the new `nsec`, the timestamp jumps backward, which is not as bad, but is still unexpected. The backward jump is limited to an order of a second, whereas the forward one is not.

Is this acceptable for `stat()`? Usually, yes, since timestamps are approximate anyway. But because KCSAN found it, it is worth investigating.

This raises a question: if there's been a recent effort to make timestamps multigrain to avoid *precisely* these issues, why isn't `shmem` using them here? I better check that.

map[ContextFile:include/linux/fs.h Name:inode]

map[Fields:[map[Name:i_mode OffsetBits:0 SizeBits:16] map[Name:i_opflags OffsetBits:16 SizeBits:16] map[Name:i_flags OffsetBits:32 SizeBits:32] map[Name:i_acl OffsetBits:64 SizeBits:64] map[Name:i_default_acl OffsetBits:128 SizeBits:64] map[Name:i_uid OffsetBits:192 SizeBits:32] map[Name:i_gid OffsetBits:224 SizeBits:32] map[Name:i_op OffsetBits:256 SizeBits:64] map[Name:i_sb OffsetBits:320 SizeBits:64] map[Name:i_mapping OffsetBits:384 SizeBits:64] map[Name:i_security OffsetBits:448 SizeBits:64] map[Name:i_ino OffsetBits:512 SizeBits:64] map[Name: OffsetBits:576 SizeBits:32] map[Name:i_rdev OffsetBits:608 SizeBits:32] map[Name:i_size OffsetBits:640 SizeBits:64] map[Name:i_atime_sec OffsetBits:704 SizeBits:64] map[Name:i_mtime_sec OffsetBits:768 SizeBits:64] map[Name:i_ctime_sec OffsetBits:832 SizeBits:64] map[Name:i_atime_nsec OffsetBits:896 SizeBits:32] map[Name:i_mtime_nsec OffsetBits:928 SizeBits:32] map[Name:i_ctime_nsec OffsetBits:960 SizeBits:32] map[Name:i_generation OffsetBits:992 SizeBits:32] map[Name:i_lock OffsetBits:1024 SizeBits:32] map[Name:i_bytes OffsetBits:1056 SizeBits:16] map[Name:i_blkbits OffsetBits:1072 SizeBits:8] map[Name:i_write_hint OffsetBits:1080 SizeBits:8] map[Name:i_blocks OffsetBits:1088 SizeBits:64] map[Name:i_state OffsetBits:1152 SizeBits:32] map[Name:i_rwsem OffsetBits:1216 SizeBits:320] map[Name:dirtied_when OffsetBits:1536 SizeBits:64] map[Name:dirtied_time_when OffsetBits:1600 SizeBits:64] map[Name:i_hash OffsetBits:1664 SizeBits:128] map[Name:i_io_list OffsetBits:1792 SizeBits:128] map[Name:i_wb OffsetBits:1920 SizeBits:64] map[Name:i_wb_frn_winner OffsetBits:1984 SizeBits:32] map[Name:i_wb_frn_avg_time OffsetBits:2016 SizeBits:16] map[Name:i_wb_frn_history OffsetBits:2032 SizeBits:16] map[Name:i_lru OffsetBits:2048 SizeBits:128] map[Name:i_sb_list OffsetBits:2176 SizeBits:128] map[Name:i_wb_list OffsetBits:2304 SizeBits:128] map[Name: OffsetBits:2432 SizeBits:128] map[Name:i_version OffsetBits:2560 SizeBits:64] map[Name:i_sequence OffsetBits:2624 SizeBits:64] map[Name:i_count OffsetBits:2688 SizeBits:32] map[Name:i_dio_count OffsetBits:2720 SizeBits:32] map[Name:i_writecount OffsetBits:2752 SizeBits:32] map[Name:i_readcount OffsetBits:2784 SizeBits:32] map[Name: OffsetBits:2816 SizeBits:64] map[Name:i_flctx OffsetBits:2880 SizeBits:64] map[Name:i_data OffsetBits:2944 SizeBits:1536] map[Name: OffsetBits:4480 SizeBits:128] map[Name: OffsetBits:4608 SizeBits:64] map[Name:i_fsnotify_mask OffsetBits:4672 SizeBits:32] map[Name:i_fsnotify_marks OffsetBits:4736 SizeBits:64] map[Name:i_private OffsetBits:4800 SizeBits:64]]]

gemini-3-flash-preview

							input: 21248
							output: 41
							thoughts: 2231
						

**Analyzing a Timestamp Data Race in the Kernel**

Okay, so I'm looking at this KCSAN report, and it's pointing to a potential data race with `i_ctime_nsec`. The offset calculation checks out; it's at 120 bytes, and `i_ctime_sec` is 104 bytes before that. The report flags `i_ctime_nsec`, so let's dig deeper.

If the "multigrain" time feature (`is_mgtime(inode)`) is enabled, `inode_set_ctime_current` uses `try_cmpxchg` on `i_ctime_nsec`, which *should* handle races for *that* field. However, even with the atomic update on `nsec`, a race could still exist with the non-atomic update of `i_ctime_sec`. But, hang on. If it *is* multigrain, then `fill_mg_cmtime` is used for reading. The code shows it reads them *separately* from the inode struct. The writer is updating `nsec` and *then* `sec`. The reader is reading `sec` and then reading `nsec`.

If the writer is between updating `nsec` and `sec`, we have a potential race! We get the "old `sec` but the new `nsec`".  If `sec` is about to increment and `nsec` is small, we could end up with an older timestamp.

For example:
- Old: `sec=10, nsec=999,000,000`
- New: `sec=11, nsec=1,000,000`
- Writer updates `nsec` to `1,000,000`
- Reader reads `sec=10`
- Reader reads `nsec=1,000,000`
- Result: `10.001` which is *less* than `10.999`.

If the Reader reads the new `sec` and new `nsec` it is correct, as expected.

However, the report says `is_mgtime` is *false*, which seems odd! Let's follow that.

If multigrain isn't active, the writer just writes `i_ctime_sec` *then* `i_ctime_nsec`, and the reader just reads `i_ctime_sec` and `i_ctime_nsec` - plain loads/stores.

If Reader(1) after Writer(1) and Reader(2) before Writer(2), we get a "newer" timestamp, which is generally worse than an older timestamp.

So we have:
- Writer: `i_ctime_sec = NEW_SEC`, then `i_ctime_nsec = NEW_NSEC`
- Reader: `read(i_ctime_sec)`, then `read(i_ctime_nsec)`

If the Reader reads NEW `sec` and OLD `nsec`, this is a future timestamp. Example 10.999 -> 11.001. So, we'd get 11.999, which is nearly a *second* in the future!

This isn't a *crash* or corruption, but it *is* a known consistency issue in the kernel. In many filesystems, non-atomic timestamp updates is known to have this issue. Layton has been doing work on multigrain timestamps, but that is mainly to solve the "same timestamp for different updates" problem.

Since KCSAN is reporting it, it's a data race. Is it benign? Not in the sense of the resulting value *always* being correct, but the memory accesses are not going to cause a crash. It is "informational".

The kernel code *tries* to be more careful with multigrain timestamps, but here we aren't using them. Looking at `inode_set_ctime_current` in the code, I see it now uses `smp_load_acquire` for the `i_ctime_nsec` *if* the inode is multigrain. If NOT multigrain, it is a plain store, and `inode_get_ctime_nsec` is a plain load! That is why KCSAN is reporting it.

To avoid the KCSAN report, we could use `READ_ONCE`/`WRITE_ONCE`. But that wouldn't fix the race with `sec`.

Wait a minute, why is `is_mgtime` false for shmem? I *thought* I saw `FS_MGTIME` in the `shmem_fs_type` definition. The stack trace says `shmem_xattr_handler_set`. *That's* definitely shmem

map[ContextFile:mm/shmem.c IncludeLines:true Name:shmem_fs_type]

map[SourceCode:5351:	static struct file_system_type shmem_fs_type = {
5352:		.owner		= THIS_MODULE,
5353:		.name		= "tmpfs",
5354:		.init_fs_context = shmem_init_fs_context,
5355:	#ifdef CONFIG_TMPFS
5356:		.parameters	= shmem_fs_parameters,
5357:	#endif
5358:		.kill_sb	= kill_anon_super,
5359:		.fs_flags	= FS_USERNS_MOUNT | FS_ALLOW_IDMAP | FS_MGTIME,
5360:	};
 SourceFile:mm/shmem.c]

gemini-3-flash-preview

							input: 24749
							output: 20
							thoughts: 385
						

**Debugging a Kernel Time Issue**

Okay, so I've got this bug report and the stack trace is pointing directly to line 2842 in `fs/inode.c`. That's where the `inode_set_ctime_to_ts()` function is called. The logic is, if `is_mgtime(inode)` is *not* true, then we set the change time. So why did it go there? If `is_mgtime()` returns true, it should have skipped past it.

Let me see. `is_mgtime(inode)` checks `inode->i_opflags` against `IOP_MGTIME`. And `IOP_MGTIME` gets set in `inode_init_always_gfp` if the filesystem type has `FS_MGTIME`. Ah, that makes sense. Is the shmem mount not setting `FS_MGTIME`? I know `shmem_fs_type` defines the mount's file system type and that's what `sb->s_type` points to. I'm going to have to check if `FS_MGTIME` is set there.

Hmm, but wait. Is it possible that `IOP_MGTIME` is *not* getting set for some reason? Or maybe the line numbers are wrong, the report's pointing me at the wrong spot. Let me confirm. Ah, it's specific, `inode_set_ctime_current+0x256/0x960 fs/inode.c:2842`. That's pretty precise.

Oh! I think I've got it. Maybe `FS_MGTIME` was added to `shmem` in some kernel versions but not others? Or, perhaps, it was added and then removed? No, wait a minute, it does look like it's there.

Okay, let's step back and re-examine `inode_set_ctime_current`. It's possible I am misinterpreting line 2842. I'm going to go back to the source code and re-read the surrounding lines in `fs/inode.c` to see if I missed something important.

map[File:fs/inode.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-only
/*
 * (C) 1997 Linus Torvalds
 * (C) 1999 Andrea Arcangeli <andrea@suse.de> (dynamic inode allocation)
 */
#include <linux/export.h>
#include <linux/fs.h>
#include <linux/filelock.h>
#include <linux/mm.h>
#include <linux/backing-dev.h>
#include <linux/hash.h>
#include <linux/swap.h>
#include <linux/security.h>
#include <linux/cdev.h>
#include <linux/memblock.h>
#include <linux/fsnotify.h>
#include <linux/fsverity.h>
#include <linux/mount.h>
#include <linux/posix_acl.h>
#include <linux/buffer_head.h> /* for inode_has_buffers */
#include <linux/ratelimit.h>
#include <linux/list_lru.h>
#include <linux/iversion.h>
#include <linux/rw_hint.h>
#include <linux/seq_file.h>
#include <linux/debugfs.h>
#include <trace/events/writeback.h>
#define CREATE_TRACE_POINTS
#include <trace/events/timestamp.h>

#include "internal.h"

/*
 * Inode locking rules:
 *
 * inode->i_lock protects:
 *   inode->i_state, inode->i_hash, __iget(), inode->i_io_list
 * Inode LRU list locks protect:
 *   inode->i_sb->s_inode_lru, inode->i_lru
 * inode->i_sb->s_inode_list_lock protects:
 *   inode->i_sb->s_inodes, inode->i_sb_list
 * bdi->wb.list_lock protects:
 *   bdi->wb.b_{dirty,io,more_io,dirty_time}, inode->i_io_list
 * inode_hash_lock protects:
 *   inode_hashtable, inode->i_hash
 *
 * Lock ordering:
 *
 * inode->i_sb->s_inode_list_lock
 *   inode->i_lock
 *     Inode LRU list locks
 *
 * bdi->wb.list_lock
 *   inode->i_lock
 *
 * inode_hash_lock
 *   inode->i_sb->s_inode_list_lock
 *   inode->i_lock
 *
 * iunique_lock
 *   inode_hash_lock
 */

static unsigned int i_hash_mask __ro_after_init;
static unsigned int i_hash_shift __ro_after_init;
static struct hlist_head *inode_hashtable __ro_after_init;
static __cacheline_aligned_in_smp DEFINE_SPINLOCK(inode_hash_lock);

/*
 * Empty aops. Can be used for the cases where the user does not
 * define any of the address_space operations.
 */
const struct address_space_operations empty_aops = {
};
EXPORT_SYMBOL(empty_aops);

static DEFINE_PER_CPU(unsigned long, nr_inodes);
static DEFINE_PER_CPU(unsigned long, nr_unused);

static struct kmem_cache *inode_cachep __ro_after_init;

static long get_nr_inodes(void)
{
	int i;
	long sum = 0;
	for_each_possible_cpu(i)
		sum += per_cpu(nr_inodes, i);
	return sum < 0 ? 0 : sum;
}

static inline long get_nr_inodes_unused(void)
{
	int i;
	long sum = 0;
	for_each_possible_cpu(i)
		sum += per_cpu(nr_unused, i);
	return sum < 0 ? 0 : sum;
}

long get_nr_dirty_inodes(void)
{
	/* not actually dirty inodes, but a wild approximation */
	long nr_dirty = get_nr_inodes() - get_nr_inodes_unused();
	return nr_dirty > 0 ? nr_dirty : 0;
}

#ifdef CONFIG_DEBUG_FS
static DEFINE_PER_CPU(long, mg_ctime_updates);
static DEFINE_PER_CPU(long, mg_fine_stamps);
static DEFINE_PER_CPU(long, mg_ctime_swaps);

static unsigned long get_mg_ctime_updates(void)
{
	unsigned long sum = 0;
	int i;

	for_each_possible_cpu(i)
		sum += data_race(per_cpu(mg_ctime_updates, i));
	return sum;
}

static unsigned long get_mg_fine_stamps(void)
{
	unsigned long sum = 0;
	int i;

	for_each_possible_cpu(i)
		sum += data_race(per_cpu(mg_fine_stamps, i));
	return sum;
}

static unsigned long get_mg_ctime_swaps(void)
{
	unsigned long sum = 0;
	int i;

	for_each_possible_cpu(i)
		sum += data_race(per_cpu(mg_ctime_swaps, i));
	return sum;
}

#define mgtime_counter_inc(__var)	this_cpu_inc(__var)

static int mgts_show(struct seq_file *s, void *p)
{
	unsigned long ctime_updates = get_mg_ctime_updates();
	unsigned long ctime_swaps = get_mg_ctime_swaps();
	unsigned long fine_stamps = get_mg_fine_stamps();
	unsigned long floor_swaps = timekeeping_get_mg_floor_swaps();

	seq_printf(s, "%lu %lu %lu %lu\n",
		   ctime_updates, ctime_swaps, fine_stamps, floor_swaps);
	return 0;
}

DEFINE_SHOW_ATTRIBUTE(mgts);

static int __init mg_debugfs_init(void)
{
	debugfs_create_file("multigrain_timestamps", S_IFREG | S_IRUGO, NULL, NULL, &mgts_fops);
	return 0;
}
late_initcall(mg_debugfs_init);

#else /* ! CONFIG_DEBUG_FS */

#define mgtime_counter_inc(__var)	do { } while (0)

#endif /* CONFIG_DEBUG_FS */

/*
 * Handle nr_inode sysctl
 */
#ifdef CONFIG_SYSCTL
/*
 * Statistics gathering..
 */
static struct inodes_stat_t inodes_stat;

static int proc_nr_inodes(const struct ctl_table *table, int write, void *buffer,
			  size_t *lenp, loff_t *ppos)
{
	inodes_stat.nr_inodes = get_nr_inodes();
	inodes_stat.nr_unused = get_nr_inodes_unused();
	return proc_doulongvec_minmax(table, write, buffer, lenp, ppos);
}

static const struct ctl_table inodes_sysctls[] = {
	{
		.procname	= "inode-nr",
		.data		= &inodes_stat,
		.maxlen		= 2*sizeof(long),
		.mode		= 0444,
		.proc_handler	= proc_nr_inodes,
	},
	{
		.procname	= "inode-state",
		.data		= &inodes_stat,
		.maxlen		= 7*sizeof(long),
		.mode		= 0444,
		.proc_handler	= proc_nr_inodes,
	},
};

static int __init init_fs_inode_sysctls(void)
{
	register_sysctl_init("fs", inodes_sysctls);
	return 0;
}
early_initcall(init_fs_inode_sysctls);
#endif

static int no_open(struct inode *inode, struct file *file)
{
	return -ENXIO;
}

/**
 * inode_init_always_gfp - perform inode structure initialisation
 * @sb: superblock inode belongs to
 * @inode: inode to initialise
 * @gfp: allocation flags
 *
 * These are initializations that need to be done on every inode
 * allocation as the fields are not initialised by slab allocation.
 * If there are additional allocations required @gfp is used.
 */
int inode_init_always_gfp(struct super_block *sb, struct inode *inode, gfp_t gfp)
{
	static const struct inode_operations empty_iops;
	static const struct file_operations no_open_fops = {.open = no_open};
	struct address_space *const mapping = &inode->i_data;

	inode->i_sb = sb;
	inode->i_blkbits = sb->s_blocksize_bits;
	inode->i_flags = 0;
	inode_state_assign_raw(inode, 0);
	atomic64_set(&inode->i_sequence, 0);
	atomic_set(&inode->i_count, 1);
	inode->i_op = &empty_iops;
	inode->i_fop = &no_open_fops;
	inode->i_ino = 0;
	inode->__i_nlink = 1;
	inode->i_opflags = 0;
	if (sb->s_xattr)
		inode->i_opflags |= IOP_XATTR;
	if (sb->s_type->fs_flags & FS_MGTIME)
		inode->i_opflags |= IOP_MGTIME;
	i_uid_write(inode, 0);
	i_gid_write(inode, 0);
	atomic_set(&inode->i_writecount, 0);
	inode->i_size = 0;
	inode->i_write_hint = WRITE_LIFE_NOT_SET;
	inode->i_blocks = 0;
	inode->i_bytes = 0;
	inode->i_generation = 0;
	inode->i_pipe = NULL;
	inode->i_cdev = NULL;
	inode->i_link = NULL;
	inode->i_dir_seq = 0;
	inode->i_rdev = 0;
	inode->dirtied_when = 0;

#ifdef CONFIG_CGROUP_WRITEBACK
	inode->i_wb_frn_winner = 0;
	inode->i_wb_frn_avg_time = 0;
	inode->i_wb_frn_history = 0;
#endif

	spin_lock_init(&inode->i_lock);
	lockdep_set_class(&inode->i_lock, &sb->s_type->i_lock_key);

	init_rwsem(&inode->i_rwsem);
	lockdep_set_class(&inode->i_rwsem, &sb->s_type->i_mutex_key);

	atomic_set(&inode->i_dio_count, 0);

	mapping->a_ops = &empty_aops;
	mapping->host = inode;
	mapping->flags = 0;
	mapping->wb_err = 0;
	atomic_set(&mapping->i_mmap_writable, 0);
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
	atomic_set(&mapping->nr_thps, 0);
#endif
	mapping_set_gfp_mask(mapping, GFP_HIGHUSER_MOVABLE);
	mapping->i_private_data = NULL;
	mapping->writeback_index = 0;
	init_rwsem(&mapping->invalidate_lock);
	lockdep_set_class_and_name(&mapping->invalidate_lock,
				   &sb->s_type->invalidate_lock_key,
				   "mapping.invalidate_lock");
	if (sb->s_iflags & SB_I_STABLE_WRITES)
		mapping_set_stable_writes(mapping);
	inode->i_private = NULL;
	inode->i_mapping = mapping;
	INIT_HLIST_HEAD(&inode->i_dentry);	/* buggered by rcu freeing */
#ifdef CONFIG_FS_POSIX_ACL
	inode->i_acl = inode->i_default_acl = ACL_NOT_CACHED;
#endif

#ifdef CONFIG_FSNOTIFY
	inode->i_fsnotify_mask = 0;
#endif
	inode->i_flctx = NULL;

	if (unlikely(security_inode_alloc(inode, gfp)))
		return -ENOMEM;

	this_cpu_inc(nr_inodes);

	return 0;
}
EXPORT_SYMBOL(inode_init_always_gfp);

void free_inode_nonrcu(struct inode *inode)
{
	kmem_cache_free(inode_cachep, inode);
}
EXPORT_SYMBOL(free_inode_nonrcu);

static void i_callback(struct rcu_head *head)
{
	struct inode *inode = container_of(head, struct inode, i_rcu);
	if (inode->free_inode)
		inode->free_inode(inode);
	else
		free_inode_nonrcu(inode);
}

/**
 *	alloc_inode 	- obtain an inode
 *	@sb: superblock
 *
 *	Allocates a new inode for given superblock.
 *	Inode wont be chained in superblock s_inodes list
 *	This means :
 *	- fs can't be unmount
 *	- quotas, fsnotify, writeback can't work
 */
struct inode *alloc_inode(struct super_block *sb)
{
	const struct super_operations *ops = sb->s_op;
	struct inode *inode;

	if (ops->alloc_inode)
		inode = ops->alloc_inode(sb);
	else
		inode = alloc_inode_sb(sb, inode_cachep, GFP_KERNEL);

	if (!inode)
		return NULL;

	if (unlikely(inode_init_always(sb, inode))) {
		if (ops->destroy_inode) {
			ops->destroy_inode(inode);
			if (!ops->free_inode)
				return NULL;
		}
		inode->free_inode = ops->free_inode;
		i_callback(&inode->i_rcu);
		return NULL;
	}

	return inode;
}

void __destroy_inode(struct inode *inode)
{
	BUG_ON(inode_has_buffers(inode));
	inode_detach_wb(inode);
	security_inode_free(inode);
	fsnotify_inode_delete(inode);
	locks_free_lock_context(inode);
	if (!inode->i_nlink) {
		WARN_ON(atomic_long_read(&inode->i_sb->s_remove_count) == 0);
		atomic_long_dec(&inode->i_sb->s_remove_count);
	}

#ifdef CONFIG_FS_POSIX_ACL
	if (inode->i_acl && !is_uncached_acl(inode->i_acl))
		posix_acl_release(inode->i_acl);
	if (inode->i_default_acl && !is_uncached_acl(inode->i_default_acl))
		posix_acl_release(inode->i_default_acl);
#endif
	this_cpu_dec(nr_inodes);
}
EXPORT_SYMBOL(__destroy_inode);

static void destroy_inode(struct inode *inode)
{
	const struct super_operations *ops = inode->i_sb->s_op;

	BUG_ON(!list_empty(&inode->i_lru));
	__destroy_inode(inode);
	if (ops->destroy_inode) {
		ops->destroy_inode(inode);
		if (!ops->free_inode)
			return;
	}
	inode->free_inode = ops->free_inode;
	call_rcu(&inode->i_rcu, i_callback);
}

/**
 * drop_nlink - directly drop an inode's link count
 * @inode: inode
 *
 * This is a low-level filesystem helper to replace any
 * direct filesystem manipulation of i_nlink.  In cases
 * where we are attempting to track writes to the
 * filesystem, a decrement to zero means an imminent
 * write when the file is truncated and actually unlinked
 * on the filesystem.
 */
void drop_nlink(struct inode *inode)
{
	WARN_ON(inode->i_nlink == 0);
	inode->__i_nlink--;
	if (!inode->i_nlink)
		atomic_long_inc(&inode->i_sb->s_remove_count);
}
EXPORT_SYMBOL(drop_nlink);

/**
 * clear_nlink - directly zero an inode's link count
 * @inode: inode
 *
 * This is a low-level filesystem helper to replace any
 * direct filesystem manipulation of i_nlink.  See
 * drop_nlink() for why we care about i_nlink hitting zero.
 */
void clear_nlink(struct inode *inode)
{
	if (inode->i_nlink) {
		inode->__i_nlink = 0;
		atomic_long_inc(&inode->i_sb->s_remove_count);
	}
}
EXPORT_SYMBOL(clear_nlink);

/**
 * set_nlink - directly set an inode's link count
 * @inode: inode
 * @nlink: new nlink (should be non-zero)
 *
 * This is a low-level filesystem helper to replace any
 * direct filesystem manipulation of i_nlink.
 */
void set_nlink(struct inode *inode, unsigned int nlink)
{
	if (!nlink) {
		clear_nlink(inode);
	} else {
		/* Yes, some filesystems do change nlink from zero to one */
		if (inode->i_nlink == 0)
			atomic_long_dec(&inode->i_sb->s_remove_count);

		inode->__i_nlink = nlink;
	}
}
EXPORT_SYMBOL(set_nlink);

/**
 * inc_nlink - directly increment an inode's link count
 * @inode: inode
 *
 * This is a low-level filesystem helper to replace any
 * direct filesystem manipulation of i_nlink.  Currently,
 * it is only here for parity with dec_nlink().
 */
void inc_nlink(struct inode *inode)
{
	if (unlikely(inode->i_nlink == 0)) {
		WARN_ON(!(inode_state_read_once(inode) & I_LINKABLE));
		atomic_long_dec(&inode->i_sb->s_remove_count);
	}

	inode->__i_nlink++;
}
EXPORT_SYMBOL(inc_nlink);

static void __address_space_init_once(struct address_space *mapping)
{
	xa_init_flags(&mapping->i_pages, XA_FLAGS_LOCK_IRQ | XA_FLAGS_ACCOUNT);
	init_rwsem(&mapping->i_mmap_rwsem);
	INIT_LIST_HEAD(&mapping->i_private_list);
	spin_lock_init(&mapping->i_private_lock);
	mapping->i_mmap = RB_ROOT_CACHED;
}

void address_space_init_once(struct address_space *mapping)
{
	memset(mapping, 0, sizeof(*mapping));
	__address_space_init_once(mapping);
}
EXPORT_SYMBOL(address_space_init_once);

/*
 * These are initializations that only need to be done
 * once, because the fields are idempotent across use
 * of the inode, so let the slab aware of that.
 */
void inode_init_once(struct inode *inode)
{
	memset(inode, 0, sizeof(*inode));
	INIT_HLIST_NODE(&inode->i_hash);
	INIT_LIST_HEAD(&inode->i_devices);
	INIT_LIST_HEAD(&inode->i_io_list);
	INIT_LIST_HEAD(&inode->i_wb_list);
	INIT_LIST_HEAD(&inode->i_lru);
	INIT_LIST_HEAD(&inode->i_sb_list);
	__address_space_init_once(&inode->i_data);
	i_size_ordered_init(inode);
}
EXPORT_SYMBOL(inode_init_once);

static void init_once(void *foo)
{
	struct inode *inode = (struct inode *) foo;

	inode_init_once(inode);
}

/*
 * get additional reference to inode; caller must already hold one.
 */
void ihold(struct inode *inode)
{
	WARN_ON(atomic_inc_return(&inode->i_count) < 2);
}
EXPORT_SYMBOL(ihold);

struct wait_queue_head *inode_bit_waitqueue(struct wait_bit_queue_entry *wqe,
					    struct inode *inode, u32 bit)
{
	void *bit_address;

	bit_address = inode_state_wait_address(inode, bit);
	init_wait_var_entry(wqe, bit_address, 0);
	return __var_waitqueue(bit_address);
}
EXPORT_SYMBOL(inode_bit_waitqueue);

void wait_on_new_inode(struct inode *inode)
{
	struct wait_bit_queue_entry wqe;
	struct wait_queue_head *wq_head;

	spin_lock(&inode->i_lock);
	if (!(inode_state_read(inode) & I_NEW)) {
		spin_unlock(&inode->i_lock);
		return;
	}

	wq_head = inode_bit_waitqueue(&wqe, inode, __I_NEW);
	for (;;) {
		prepare_to_wait_event(wq_head, &wqe.wq_entry, TASK_UNINTERRUPTIBLE);
		if (!(inode_state_read(inode) & I_NEW))
			break;
		spin_unlock(&inode->i_lock);
		schedule();
		spin_lock(&inode->i_lock);
	}
	finish_wait(wq_head, &wqe.wq_entry);
	WARN_ON(inode_state_read(inode) & I_NEW);
	spin_unlock(&inode->i_lock);
}
EXPORT_SYMBOL(wait_on_new_inode);

static void __inode_lru_list_add(struct inode *inode, bool rotate)
{
	lockdep_assert_held(&inode->i_lock);

	if (inode_state_read(inode) & (I_DIRTY_ALL | I_SYNC | I_FREEING | I_WILL_FREE))
		return;
	if (icount_read(inode))
		return;
	if (!(inode->i_sb->s_flags & SB_ACTIVE))
		return;
	if (!mapping_shrinkable(&inode->i_data))
		return;

	if (list_lru_add_obj(&inode->i_sb->s_inode_lru, &inode->i_lru))
		this_cpu_inc(nr_unused);
	else if (rotate)
		inode_state_set(inode, I_REFERENCED);
}

/*
 * Add inode to LRU if needed (inode is unused and clean).
 */
void inode_lru_list_add(struct inode *inode)
{
	__inode_lru_list_add(inode, false);
}

static void inode_lru_list_del(struct inode *inode)
{
	if (list_empty(&inode->i_lru))
		return;

	if (list_lru_del_obj(&inode->i_sb->s_inode_lru, &inode->i_lru))
		this_cpu_dec(nr_unused);
}

static void inode_pin_lru_isolating(struct inode *inode)
{
	lockdep_assert_held(&inode->i_lock);
	WARN_ON(inode_state_read(inode) & (I_LRU_ISOLATING | I_FREEING | I_WILL_FREE));
	inode_state_set(inode, I_LRU_ISOLATING);
}

static void inode_unpin_lru_isolating(struct inode *inode)
{
	spin_lock(&inode->i_lock);
	WARN_ON(!(inode_state_read(inode) & I_LRU_ISOLATING));
	inode_state_clear(inode, I_LRU_ISOLATING);
	/* Called with inode->i_lock which ensures memory ordering. */
	inode_wake_up_bit(inode, __I_LRU_ISOLATING);
	spin_unlock(&inode->i_lock);
}

static void inode_wait_for_lru_isolating(struct inode *inode)
{
	struct wait_bit_queue_entry wqe;
	struct wait_queue_head *wq_head;

	lockdep_assert_held(&inode->i_lock);
	if (!(inode_state_read(inode) & I_LRU_ISOLATING))
		return;

	wq_head = inode_bit_waitqueue(&wqe, inode, __I_LRU_ISOLATING);
	for (;;) {
		prepare_to_wait_event(wq_head, &wqe.wq_entry, TASK_UNINTERRUPTIBLE);
		/*
		 * Checking I_LRU_ISOLATING with inode->i_lock guarantees
		 * memory ordering.
		 */
		if (!(inode_state_read(inode) & I_LRU_ISOLATING))
			break;
		spin_unlock(&inode->i_lock);
		schedule();
		spin_lock(&inode->i_lock);
	}
	finish_wait(wq_head, &wqe.wq_entry);
	WARN_ON(inode_state_read(inode) & I_LRU_ISOLATING);
}

/**
 * inode_sb_list_add - add inode to the superblock list of inodes
 * @inode: inode to add
 */
void inode_sb_list_add(struct inode *inode)
{
	struct super_block *sb = inode->i_sb;

	spin_lock(&sb->s_inode_list_lock);
	list_add(&inode->i_sb_list, &sb->s_inodes);
	spin_unlock(&sb->s_inode_list_lock);
}
EXPORT_SYMBOL_GPL(inode_sb_list_add);

static inline void inode_sb_list_del(struct inode *inode)
{
	struct super_block *sb = inode->i_sb;

	if (!list_empty(&inode->i_sb_list)) {
		spin_lock(&sb->s_inode_list_lock);
		list_del_init(&inode->i_sb_list);
		spin_unlock(&sb->s_inode_list_lock);
	}
}

static unsigned long hash(struct super_block *sb, unsigned long hashval)
{
	unsigned long tmp;

	tmp = (hashval * (unsigned long)sb) ^ (GOLDEN_RATIO_PRIME + hashval) /
			L1_CACHE_BYTES;
	tmp = tmp ^ ((tmp ^ GOLDEN_RATIO_PRIME) >> i_hash_shift);
	return tmp & i_hash_mask;
}

/**
 *	__insert_inode_hash - hash an inode
 *	@inode: unhashed inode
 *	@hashval: unsigned long value used to locate this object in the
 *		inode_hashtable.
 *
 *	Add an inode to the inode hash for this superblock.
 */
void __insert_inode_hash(struct inode *inode, unsigned long hashval)
{
	struct hlist_head *b = inode_hashtable + hash(inode->i_sb, hashval);

	spin_lock(&inode_hash_lock);
	spin_lock(&inode->i_lock);
	hlist_add_head_rcu(&inode->i_hash, b);
	spin_unlock(&inode->i_lock);
	spin_unlock(&inode_hash_lock);
}
EXPORT_SYMBOL(__insert_inode_hash);

/**
 *	__remove_inode_hash - remove an inode from the hash
 *	@inode: inode to unhash
 *
 *	Remove an inode from the superblock.
 */
void __remove_inode_hash(struct inode *inode)
{
	spin_lock(&inode_hash_lock);
	spin_lock(&inode->i_lock);
	hlist_del_init_rcu(&inode->i_hash);
	spin_unlock(&inode->i_lock);
	spin_unlock(&inode_hash_lock);
}
EXPORT_SYMBOL(__remove_inode_hash);

void dump_mapping(const struct address_space *mapping)
{
	struct inode *host;
	const struct address_space_operations *a_ops;
	struct hlist_node *dentry_first;
	struct dentry *dentry_ptr;
	struct dentry dentry;
	char fname[64] = {};
	unsigned long ino;

	/*
	 * If mapping is an invalid pointer, we don't want to crash
	 * accessing it, so probe everything depending on it carefully.
	 */
	if (get_kernel_nofault(host, &mapping->host) ||
	    get_kernel_nofault(a_ops, &mapping->a_ops)) {
		pr_warn("invalid mapping:%px\n", mapping);
		return;
	}

	if (!host) {
		pr_warn("aops:%ps\n", a_ops);
		return;
	}

	if (get_kernel_nofault(dentry_first, &host->i_dentry.first) ||
	    get_kernel_nofault(ino, &host->i_ino)) {
		pr_warn("aops:%ps invalid inode:%px\n", a_ops, host);
		return;
	}

	if (!dentry_first) {
		pr_warn("aops:%ps ino:%lx\n", a_ops, ino);
		return;
	}

	dentry_ptr = container_of(dentry_first, struct dentry, d_u.d_alias);
	if (get_kernel_nofault(dentry, dentry_ptr) ||
	    !dentry.d_parent || !dentry.d_name.name) {
		pr_warn("aops:%ps ino:%lx invalid dentry:%px\n",
				a_ops, ino, dentry_ptr);
		return;
	}

	if (strncpy_from_kernel_nofault(fname, dentry.d_name.name, 63) < 0)
		strscpy(fname, "<invalid>");
	/*
	 * Even if strncpy_from_kernel_nofault() succeeded,
	 * the fname could be unreliable
	 */
	pr_warn("aops:%ps ino:%lx dentry name(?):\"%s\"\n",
		a_ops, ino, fname);
}

void clear_inode(struct inode *inode)
{
	/*
	 * Only IS_VERITY() inodes can have verity info, so start by checking
	 * for IS_VERITY() (which is faster than retrieving the pointer to the
	 * verity info).  This minimizes overhead for non-verity inodes.
	 */
	if (IS_ENABLED(CONFIG_FS_VERITY) && IS_VERITY(inode))
		fsverity_cleanup_inode(inode);

	/*
	 * We have to cycle the i_pages lock here because reclaim can be in the
	 * process of removing the last page (in __filemap_remove_folio())
	 * and we must not free the mapping under it.
	 */
	xa_lock_irq(&inode->i_data.i_pages);
	BUG_ON(inode->i_data.nrpages);
	/*
	 * Almost always, mapping_empty(&inode->i_data) here; but there are
	 * two known and long-standing ways in which nodes may get left behind
	 * (when deep radix-tree node allocation failed partway; or when THP
	 * collapse_file() failed). Until those two known cases are cleaned up,
	 * or a cleanup function is called here, do not BUG_ON(!mapping_empty),
	 * nor even WARN_ON(!mapping_empty).
	 */
	xa_unlock_irq(&inode->i_data.i_pages);
	BUG_ON(!list_empty(&inode->i_data.i_private_list));
	BUG_ON(!(inode_state_read_once(inode) & I_FREEING));
	BUG_ON(inode_state_read_once(inode) & I_CLEAR);
	BUG_ON(!list_empty(&inode->i_wb_list));
	/* don't need i_lock here, no concurrent mods to i_state */
	inode_state_assign_raw(inode, I_FREEING | I_CLEAR);
}
EXPORT_SYMBOL(clear_inode);

/*
 * Free the inode passed in, removing it from the lists it is still connected
 * to. We remove any pages still attached to the inode and wait for any IO that
 * is still in progress before finally destroying the inode.
 *
 * An inode must already be marked I_FREEING so that we avoid the inode being
 * moved back onto lists if we race with other code that manipulates the lists
 * (e.g. writeback_single_inode). The caller is responsible for setting this.
 *
 * An inode must already be removed from the LRU list before being evicted from
 * the cache. This should occur atomically with setting the I_FREEING state
 * flag, so no inodes here should ever be on the LRU when being evicted.
 */
static void evict(struct inode *inode)
{
	const struct super_operations *op = inode->i_sb->s_op;

	BUG_ON(!(inode_state_read_once(inode) & I_FREEING));
	BUG_ON(!list_empty(&inode->i_lru));

	inode_io_list_del(inode);
	inode_sb_list_del(inode);

	spin_lock(&inode->i_lock);
	inode_wait_for_lru_isolating(inode);

	/*
	 * Wait for flusher thread to be done with the inode so that filesystem
	 * does not start destroying it while writeback is still running. Since
	 * the inode has I_FREEING set, flusher thread won't start new work on
	 * the inode.  We just have to wait for running writeback to finish.
	 */
	inode_wait_for_writeback(inode);
	spin_unlock(&inode->i_lock);

	if (op->evict_inode) {
		op->evict_inode(inode);
	} else {
		truncate_inode_pages_final(&inode->i_data);
		clear_inode(inode);
	}
	if (S_ISCHR(inode->i_mode) && inode->i_cdev)
		cd_forget(inode);

	remove_inode_hash(inode);

	/*
	 * Wake up waiters in __wait_on_freeing_inode().
	 *
	 * It is an invariant that any thread we need to wake up is already
	 * accounted for before remove_inode_hash() acquires ->i_lock -- both
	 * sides take the lock and sleep is aborted if the inode is found
	 * unhashed. Thus either the sleeper wins and goes off CPU, or removal
	 * wins and the sleeper aborts after testing with the lock.
	 *
	 * This also means we don't need any fences for the call below.
	 */
	inode_wake_up_bit(inode, __I_NEW);
	BUG_ON(inode_state_read_once(inode) != (I_FREEING | I_CLEAR));

	destroy_inode(inode);
}

/*
 * dispose_list - dispose of the contents of a local list
 * @head: the head of the list to free
 *
 * Dispose-list gets a local list with local inodes in it, so it doesn't
 * need to worry about list corruption and SMP locks.
 */
static void dispose_list(struct list_head *head)
{
	while (!list_empty(head)) {
		struct inode *inode;

		inode = list_first_entry(head, struct inode, i_lru);
		list_del_init(&inode->i_lru);

		evict(inode);
		cond_resched();
	}
}

/**
 * evict_inodes	- evict all evictable inodes for a superblock
 * @sb:		superblock to operate on
 *
 * Make sure that no inodes with zero refcount are retained.  This is
 * called by superblock shutdown after having SB_ACTIVE flag removed,
 * so any inode reaching zero refcount during or after that call will
 * be immediately evicted.
 */
void evict_inodes(struct super_block *sb)
{
	struct inode *inode;
	LIST_HEAD(dispose);

again:
	spin_lock(&sb->s_inode_list_lock);
	list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
		if (icount_read(inode))
			continue;

		spin_lock(&inode->i_lock);
		if (icount_read(inode)) {
			spin_unlock(&inode->i_lock);
			continue;
		}
		if (inode_state_read(inode) & (I_NEW | I_FREEING | I_WILL_FREE)) {
			spin_unlock(&inode->i_lock);
			continue;
		}

		inode_state_set(inode, I_FREEING);
		inode_lru_list_del(inode);
		spin_unlock(&inode->i_lock);
		list_add(&inode->i_lru, &dispose);

		/*
		 * We can have a ton of inodes to evict at unmount time given
		 * enough memory, check to see if we need to go to sleep for a
		 * bit so we don't livelock.
		 */
		if (need_resched()) {
			spin_unlock(&sb->s_inode_list_lock);
			cond_resched();
			dispose_list(&dispose);
			goto again;
		}
	}
	spin_unlock(&sb->s_inode_list_lock);

	dispose_list(&dispose);
}
EXPORT_SYMBOL_GPL(evict_inodes);

/*
 * Isolate the inode from the LRU in preparation for freeing it.
 *
 * If the inode has the I_REFERENCED flag set, then it means that it has been
 * used recently - the flag is set in iput_final(). When we encounter such an
 * inode, clear the flag and move it to the back of the LRU so it gets another
 * pass through the LRU before it gets reclaimed. This is necessary because of
 * the fact we are doing lazy LRU updates to minimise lock contention so the
 * LRU does not have strict ordering. Hence we don't want to reclaim inodes
 * with this flag set because they are the inodes that are out of order.
 */
static enum lru_status inode_lru_isolate(struct list_head *item,
		struct list_lru_one *lru, void *arg)
{
	struct list_head *freeable = arg;
	struct inode	*inode = container_of(item, struct inode, i_lru);

	/*
	 * We are inverting the lru lock/inode->i_lock here, so use a
	 * trylock. If we fail to get the lock, just skip it.
	 */
	if (!spin_trylock(&inode->i_lock))
		return LRU_SKIP;

	/*
	 * Inodes can get referenced, redirtied, or repopulated while
	 * they're already on the LRU, and this can make them
	 * unreclaimable for a while. Remove them lazily here; iput,
	 * sync, or the last page cache deletion will requeue them.
	 */
	if (icount_read(inode) ||
	    (inode_state_read(inode) & ~I_REFERENCED) ||
	    !mapping_shrinkable(&inode->i_data)) {
		list_lru_isolate(lru, &inode->i_lru);
		spin_unlock(&inode->i_lock);
		this_cpu_dec(nr_unused);
		return LRU_REMOVED;
	}

	/* Recently referenced inodes get one more pass */
	if (inode_state_read(inode) & I_REFERENCED) {
		inode_state_clear(inode, I_REFERENCED);
		spin_unlock(&inode->i_lock);
		return LRU_ROTATE;
	}

	/*
	 * On highmem systems, mapping_shrinkable() permits dropping
	 * page cache in order to free up struct inodes: lowmem might
	 * be under pressure before the cache inside the highmem zone.
	 */
	if (inode_has_buffers(inode) || !mapping_empty(&inode->i_data)) {
		inode_pin_lru_isolating(inode);
		spin_unlock(&inode->i_lock);
		spin_unlock(&lru->lock);
		if (remove_inode_buffers(inode)) {
			unsigned long reap;
			reap = invalidate_mapping_pages(&inode->i_data, 0, -1);
			if (current_is_kswapd())
				__count_vm_events(KSWAPD_INODESTEAL, reap);
			else
				__count_vm_events(PGINODESTEAL, reap);
			mm_account_reclaimed_pages(reap);
		}
		inode_unpin_lru_isolating(inode);
		return LRU_RETRY;
	}

	WARN_ON(inode_state_read(inode) & I_NEW);
	inode_state_set(inode, I_FREEING);
	list_lru_isolate_move(lru, &inode->i_lru, freeable);
	spin_unlock(&inode->i_lock);

	this_cpu_dec(nr_unused);
	return LRU_REMOVED;
}

/*
 * Walk the superblock inode LRU for freeable inodes and attempt to free them.
 * This is called from the superblock shrinker function with a number of inodes
 * to trim from the LRU. Inodes to be freed are moved to a temporary list and
 * then are freed outside inode_lock by dispose_list().
 */
long prune_icache_sb(struct super_block *sb, struct shrink_control *sc)
{
	LIST_HEAD(freeable);
	long freed;

	freed = list_lru_shrink_walk(&sb->s_inode_lru, sc,
				     inode_lru_isolate, &freeable);
	dispose_list(&freeable);
	return freed;
}

static void __wait_on_freeing_inode(struct inode *inode, bool hash_locked, bool rcu_locked);

/*
 * Called with the inode lock held.
 */
static struct inode *find_inode(struct super_block *sb,
				struct hlist_head *head,
				int (*test)(struct inode *, void *),
				void *data, bool hash_locked,
				bool *isnew)
{
	struct inode *inode = NULL;

	if (hash_locked)
		lockdep_assert_held(&inode_hash_lock);
	else
		lockdep_assert_not_held(&inode_hash_lock);

	rcu_read_lock();
repeat:
	hlist_for_each_entry_rcu(inode, head, i_hash) {
		if (inode->i_sb != sb)
			continue;
		if (!test(inode, data))
			continue;
		spin_lock(&inode->i_lock);
		if (inode_state_read(inode) & (I_FREEING | I_WILL_FREE)) {
			__wait_on_freeing_inode(inode, hash_locked, true);
			goto repeat;
		}
		if (unlikely(inode_state_read(inode) & I_CREATING)) {
			spin_unlock(&inode->i_lock);
			rcu_read_unlock();
			return ERR_PTR(-ESTALE);
		}
		__iget(inode);
		*isnew = !!(inode_state_read(inode) & I_NEW);
		spin_unlock(&inode->i_lock);
		rcu_read_unlock();
		return inode;
	}
	rcu_read_unlock();
	return NULL;
}

/*
 * find_inode_fast is the fast path version of find_inode, see the comment at
 * iget_locked for details.
 */
static struct inode *find_inode_fast(struct super_block *sb,
				struct hlist_head *head, unsigned long ino,
				bool hash_locked, bool *isnew)
{
	struct inode *inode = NULL;

	if (hash_locked)
		lockdep_assert_held(&inode_hash_lock);
	else
		lockdep_assert_not_held(&inode_hash_lock);

	rcu_read_lock();
repeat:
	hlist_for_each_entry_rcu(inode, head, i_hash) {
		if (inode->i_ino != ino)
			continue;
		if (inode->i_sb != sb)
			continue;
		spin_lock(&inode->i_lock);
		if (inode_state_read(inode) & (I_FREEING | I_WILL_FREE)) {
			__wait_on_freeing_inode(inode, hash_locked, true);
			goto repeat;
		}
		if (unlikely(inode_state_read(inode) & I_CREATING)) {
			spin_unlock(&inode->i_lock);
			rcu_read_unlock();
			return ERR_PTR(-ESTALE);
		}
		__iget(inode);
		*isnew = !!(inode_state_read(inode) & I_NEW);
		spin_unlock(&inode->i_lock);
		rcu_read_unlock();
		return inode;
	}
	rcu_read_unlock();
	return NULL;
}

/*
 * Each cpu owns a range of LAST_INO_BATCH numbers.
 * 'shared_last_ino' is dirtied only once out of LAST_INO_BATCH allocations,
 * to renew the exhausted range.
 *
 * This does not significantly increase overflow rate because every CPU can
 * consume at most LAST_INO_BATCH-1 unused inode numbers. So there is
 * NR_CPUS*(LAST_INO_BATCH-1) wastage. At 4096 and 1024, this is ~0.1% of the
 * 2^32 range, and is a worst-case. Even a 50% wastage would only increase
 * overflow rate by 2x, which does not seem too significant.
 *
 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
 * error if st_ino won't fit in target struct field. Use 32bit counter
 * here to attempt to avoid that.
 */
#define LAST_INO_BATCH 1024
static DEFINE_PER_CPU(unsigned int, last_ino);

unsigned int get_next_ino(void)
{
	unsigned int *p = &get_cpu_var(last_ino);
	unsigned int res = *p;

#ifdef CONFIG_SMP
	if (unlikely((res & (LAST_INO_BATCH-1)) == 0)) {
		static atomic_t shared_last_ino;
		int next = atomic_add_return(LAST_INO_BATCH, &shared_last_ino);

		res = next - LAST_INO_BATCH;
	}
#endif

	res++;
	/* get_next_ino should not provide a 0 inode number */
	if (unlikely(!res))
		res++;
	*p = res;
	put_cpu_var(last_ino);
	return res;
}
EXPORT_SYMBOL(get_next_ino);

/**
 *	new_inode 	- obtain an inode
 *	@sb: superblock
 *
 *	Allocates a new inode for given superblock. The default gfp_mask
 *	for allocations related to inode->i_mapping is GFP_HIGHUSER_MOVABLE.
 *	If HIGHMEM pages are unsuitable or it is known that pages allocated
 *	for the page cache are not reclaimable or migratable,
 *	mapping_set_gfp_mask() must be called with suitable flags on the
 *	newly created inode's mapping
 *
 */
struct inode *new_inode(struct super_block *sb)
{
	struct inode *inode;

	inode = alloc_inode(sb);
	if (inode)
		inode_sb_list_add(inode);
	return inode;
}
EXPORT_SYMBOL(new_inode);

#ifdef CONFIG_DEBUG_LOCK_ALLOC
void lockdep_annotate_inode_mutex_key(struct inode *inode)
{
	if (S_ISDIR(inode->i_mode)) {
		struct file_system_type *type = inode->i_sb->s_type;

		/* Set new key only if filesystem hasn't already changed it */
		if (lockdep_match_class(&inode->i_rwsem, &type->i_mutex_key)) {
			/*
			 * ensure nobody is actually holding i_rwsem
			 */
			init_rwsem(&inode->i_rwsem);
			lockdep_set_class(&inode->i_rwsem,
					  &type->i_mutex_dir_key);
		}
	}
}
EXPORT_SYMBOL(lockdep_annotate_inode_mutex_key);
#endif

/**
 * unlock_new_inode - clear the I_NEW state and wake up any waiters
 * @inode:	new inode to unlock
 *
 * Called when the inode is fully initialised to clear the new state of the
 * inode and wake up anyone waiting for the inode to finish initialisation.
 */
void unlock_new_inode(struct inode *inode)
{
	lockdep_annotate_inode_mutex_key(inode);
	spin_lock(&inode->i_lock);
	WARN_ON(!(inode_state_read(inode) & I_NEW));
	inode_state_clear(inode, I_NEW | I_CREATING);
	inode_wake_up_bit(inode, __I_NEW);
	spin_unlock(&inode->i_lock);
}
EXPORT_SYMBOL(unlock_new_inode);

void discard_new_inode(struct inode *inode)
{
	lockdep_annotate_inode_mutex_key(inode);
	spin_lock(&inode->i_lock);
	WARN_ON(!(inode_state_read(inode) & I_NEW));
	inode_state_clear(inode, I_NEW);
	inode_wake_up_bit(inode, __I_NEW);
	spin_unlock(&inode->i_lock);
	iput(inode);
}
EXPORT_SYMBOL(discard_new_inode);

/**
 * lock_two_nondirectories - take two i_mutexes on non-directory objects
 *
 * Lock any non-NULL argument. Passed objects must not be directories.
 * Zero, one or two objects may be locked by this function.
 *
 * @inode1: first inode to lock
 * @inode2: second inode to lock
 */
void lock_two_nondirectories(struct inode *inode1, struct inode *inode2)
{
	if (inode1)
		WARN_ON_ONCE(S_ISDIR(inode1->i_mode));
	if (inode2)
		WARN_ON_ONCE(S_ISDIR(inode2->i_mode));
	if (inode1 > inode2)
		swap(inode1, inode2);
	if (inode1)
		inode_lock(inode1);
	if (inode2 && inode2 != inode1)
		inode_lock_nested(inode2, I_MUTEX_NONDIR2);
}
EXPORT_SYMBOL(lock_two_nondirectories);

/**
 * unlock_two_nondirectories - release locks from lock_two_nondirectories()
 * @inode1: first inode to unlock
 * @inode2: second inode to unlock
 */
void unlock_two_nondirectories(struct inode *inode1, struct inode *inode2)
{
	if (inode1) {
		WARN_ON_ONCE(S_ISDIR(inode1->i_mode));
		inode_unlock(inode1);
	}
	if (inode2 && inode2 != inode1) {
		WARN_ON_ONCE(S_ISDIR(inode2->i_mode));
		inode_unlock(inode2);
	}
}
EXPORT_SYMBOL(unlock_two_nondirectories);

/**
 * inode_insert5 - obtain an inode from a mounted file system
 * @inode:	pre-allocated inode to use for insert to cache
 * @hashval:	hash value (usually inode number) to get
 * @test:	callback used for comparisons between inodes
 * @set:	callback used to initialize a new struct inode
 * @data:	opaque data pointer to pass to @test and @set
 * @isnew:	pointer to a bool which will indicate whether I_NEW is set
 *
 * Search for the inode specified by @hashval and @data in the inode cache,
 * and if present return it with an increased reference count. This is a
 * variant of iget5_locked() that doesn't allocate an inode.
 *
 * If the inode is not present in the cache, insert the pre-allocated inode and
 * return it locked, hashed, and with the I_NEW flag set. The file system gets
 * to fill it in before unlocking it via unlock_new_inode().
 *
 * Note that both @test and @set are called with the inode_hash_lock held, so
 * they can't sleep.
 */
struct inode *inode_insert5(struct inode *inode, unsigned long hashval,
			    int (*test)(struct inode *, void *),
			    int (*set)(struct inode *, void *), void *data)
{
	struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
	struct inode *old;
	bool isnew;

	might_sleep();

again:
	spin_lock(&inode_hash_lock);
	old = find_inode(inode->i_sb, head, test, data, true, &isnew);
	if (unlikely(old)) {
		/*
		 * Uhhuh, somebody else created the same inode under us.
		 * Use the old inode instead of the preallocated one.
		 */
		spin_unlock(&inode_hash_lock);
		if (IS_ERR(old))
			return NULL;
		if (unlikely(isnew))
			wait_on_new_inode(old);
		if (unlikely(inode_unhashed(old))) {
			iput(old);
			goto again;
		}
		return old;
	}

	if (set && unlikely(set(inode, data))) {
		spin_unlock(&inode_hash_lock);
		return NULL;
	}

	/*
	 * Return the locked inode with I_NEW set, the
	 * caller is responsible for filling in the contents
	 */
	spin_lock(&inode->i_lock);
	inode_state_set(inode, I_NEW);
	hlist_add_head_rcu(&inode->i_hash, head);
	spin_unlock(&inode->i_lock);

	spin_unlock(&inode_hash_lock);

	/*
	 * Add inode to the sb list if it's not already. It has I_NEW at this
	 * point, so it should be safe to test i_sb_list locklessly.
	 */
	if (list_empty(&inode->i_sb_list))
		inode_sb_list_add(inode);

	return inode;
}
EXPORT_SYMBOL(inode_insert5);

/**
 * iget5_locked - obtain an inode from a mounted file system
 * @sb:		super block of file system
 * @hashval:	hash value (usually inode number) to get
 * @test:	callback used for comparisons between inodes
 * @set:	callback used to initialize a new struct inode
 * @data:	opaque data pointer to pass to @test and @set
 *
 * Search for the inode specified by @hashval and @data in the inode cache,
 * and if present return it with an increased reference count. This is a
 * generalized version of iget_locked() for file systems where the inode
 * number is not sufficient for unique identification of an inode.
 *
 * If the inode is not present in the cache, allocate and insert a new inode
 * and return it locked, hashed, and with the I_NEW flag set. The file system
 * gets to fill it in before unlocking it via unlock_new_inode().
 *
 * Note that both @test and @set are called with the inode_hash_lock held, so
 * they can't sleep.
 */
struct inode *iget5_locked(struct super_block *sb, unsigned long hashval,
		int (*test)(struct inode *, void *),
		int (*set)(struct inode *, void *), void *data)
{
	struct inode *inode = ilookup5(sb, hashval, test, data);

	if (!inode) {
		struct inode *new = alloc_inode(sb);

		if (new) {
			inode = inode_insert5(new, hashval, test, set, data);
			if (unlikely(inode != new))
				destroy_inode(new);
		}
	}
	return inode;
}
EXPORT_SYMBOL(iget5_locked);

/**
 * iget5_locked_rcu - obtain an inode from a mounted file system
 * @sb:		super block of file system
 * @hashval:	hash value (usually inode number) to get
 * @test:	callback used for comparisons between inodes
 * @set:	callback used to initialize a new struct inode
 * @data:	opaque data pointer to pass to @test and @set
 *
 * This is equivalent to iget5_locked, except the @test callback must
 * tolerate the inode not being stable, including being mid-teardown.
 */
struct inode *iget5_locked_rcu(struct super_block *sb, unsigned long hashval,
		int (*test)(struct inode *, void *),
		int (*set)(struct inode *, void *), void *data)
{
	struct hlist_head *head = inode_hashtable + hash(sb, hashval);
	struct inode *inode, *new;
	bool isnew;

	might_sleep();

again:
	inode = find_inode(sb, head, test, data, false, &isnew);
	if (inode) {
		if (IS_ERR(inode))
			return NULL;
		if (unlikely(isnew))
			wait_on_new_inode(inode);
		if (unlikely(inode_unhashed(inode))) {
			iput(inode);
			goto again;
		}
		return inode;
	}

	new = alloc_inode(sb);
	if (new) {
		inode = inode_insert5(new, hashval, test, set, data);
		if (unlikely(inode != new))
			destroy_inode(new);
	}
	return inode;
}
EXPORT_SYMBOL_GPL(iget5_locked_rcu);

/**
 * iget_locked - obtain an inode from a mounted file system
 * @sb:		super block of file system
 * @ino:	inode number to get
 *
 * Search for the inode specified by @ino in the inode cache and if present
 * return it with an increased reference count. This is for file systems
 * where the inode number is sufficient for unique identification of an inode.
 *
 * If the inode is not in cache, allocate a new inode and return it locked,
 * hashed, and with the I_NEW flag set.  The file system gets to fill it in
 * before unlocking it via unlock_new_inode().
 */
struct inode *iget_locked(struct super_block *sb, unsigned long ino)
{
	struct hlist_head *head = inode_hashtable + hash(sb, ino);
	struct inode *inode;
	bool isnew;

	might_sleep();

again:
	inode = find_inode_fast(sb, head, ino, false, &isnew);
	if (inode) {
		if (IS_ERR(inode))
			return NULL;
		if (unlikely(isnew))
			wait_on_new_inode(inode);
		if (unlikely(inode_unhashed(inode))) {
			iput(inode);
			goto again;
		}
		return inode;
	}

	inode = alloc_inode(sb);
	if (inode) {
		struct inode *old;

		spin_lock(&inode_hash_lock);
		/* We released the lock, so.. */
		old = find_inode_fast(sb, head, ino, true, &isnew);
		if (!old) {
			inode->i_ino = ino;
			spin_lock(&inode->i_lock);
			inode_state_assign(inode, I_NEW);
			hlist_add_head_rcu(&inode->i_hash, head);
			spin_unlock(&inode->i_lock);
			spin_unlock(&inode_hash_lock);
			inode_sb_list_add(inode);

			/* Return the locked inode with I_NEW set, the
			 * caller is responsible for filling in the contents
			 */
			return inode;
		}

		/*
		 * Uhhuh, somebody else created the same inode under
		 * us. Use the old inode instead of the one we just
		 * allocated.
		 */
		spin_unlock(&inode_hash_lock);
		destroy_inode(inode);
		if (IS_ERR(old))
			return NULL;
		inode = old;
		if (unlikely(isnew))
			wait_on_new_inode(inode);
		if (unlikely(inode_unhashed(inode))) {
			iput(inode);
			goto again;
		}
	}
	return inode;
}
EXPORT_SYMBOL(iget_locked);

/*
 * search the inode cache for a matching inode number.
 * If we find one, then the inode number we are trying to
 * allocate is not unique and so we should not use it.
 *
 * Returns 1 if the inode number is unique, 0 if it is not.
 */
static int test_inode_iunique(struct super_block *sb, unsigned long ino)
{
	struct hlist_head *b = inode_hashtable + hash(sb, ino);
	struct inode *inode;

	hlist_for_each_entry_rcu(inode, b, i_hash) {
		if (inode->i_ino == ino && inode->i_sb == sb)
			return 0;
	}
	return 1;
}

/**
 *	iunique - get a unique inode number
 *	@sb: superblock
 *	@max_reserved: highest reserved inode number
 *
 *	Obtain an inode number that is unique on the system for a given
 *	superblock. This is used by file systems that have no natural
 *	permanent inode numbering system. An inode number is returned that
 *	is higher than the reserved limit but unique.
 *
 *	BUGS:
 *	With a large number of inodes live on the file system this function
 *	currently becomes quite slow.
 */
ino_t iunique(struct super_block *sb, ino_t max_reserved)
{
	/*
	 * On a 32bit, non LFS stat() call, glibc will generate an EOVERFLOW
	 * error if st_ino won't fit in target struct field. Use 32bit counter
	 * here to attempt to avoid that.
	 */
	static DEFINE_SPINLOCK(iunique_lock);
	static unsigned int counter;
	ino_t res;

	rcu_read_lock();
	spin_lock(&iunique_lock);
	do {
		if (counter <= max_reserved)
			counter = max_reserved + 1;
		res = counter++;
	} while (!test_inode_iunique(sb, res));
	spin_unlock(&iunique_lock);
	rcu_read_unlock();

	return res;
}
EXPORT_SYMBOL(iunique);

struct inode *igrab(struct inode *inode)
{
	spin_lock(&inode->i_lock);
	if (!(inode_state_read(inode) & (I_FREEING | I_WILL_FREE))) {
		__iget(inode);
		spin_unlock(&inode->i_lock);
	} else {
		spin_unlock(&inode->i_lock);
		/*
		 * Handle the case where s_op->clear_inode is not been
		 * called yet, and somebody is calling igrab
		 * while the inode is getting freed.
		 */
		inode = NULL;
	}
	return inode;
}
EXPORT_SYMBOL(igrab);

/**
 * ilookup5_nowait - search for an inode in the inode cache
 * @sb:		super block of file system to search
 * @hashval:	hash value (usually inode number) to search for
 * @test:	callback used for comparisons between inodes
 * @data:	opaque data pointer to pass to @test
 * @isnew:	return argument telling whether I_NEW was set when
 *		the inode was found in hash (the caller needs to
 *		wait for I_NEW to clear)
 *
 * Search for the inode specified by @hashval and @data in the inode cache.
 * If the inode is in the cache, the inode is returned with an incremented
 * reference count.
 *
 * Note: I_NEW is not waited upon so you have to be very careful what you do
 * with the returned inode.  You probably should be using ilookup5() instead.
 *
 * Note2: @test is called with the inode_hash_lock held, so can't sleep.
 */
struct inode *ilookup5_nowait(struct super_block *sb, unsigned long hashval,
		int (*test)(struct inode *, void *), void *data, bool *isnew)
{
	struct hlist_head *head = inode_hashtable + hash(sb, hashval);
	struct inode *inode;

	spin_lock(&inode_hash_lock);
	inode = find_inode(sb, head, test, data, true, isnew);
	spin_unlock(&inode_hash_lock);

	return IS_ERR(inode) ? NULL : inode;
}
EXPORT_SYMBOL(ilookup5_nowait);

/**
 * ilookup5 - search for an inode in the inode cache
 * @sb:		super block of file system to search
 * @hashval:	hash value (usually inode number) to search for
 * @test:	callback used for comparisons between inodes
 * @data:	opaque data pointer to pass to @test
 *
 * Search for the inode specified by @hashval and @data in the inode cache,
 * and if the inode is in the cache, return the inode with an incremented
 * reference count.  Waits on I_NEW before returning the inode.
 * returned with an incremented reference count.
 *
 * This is a generalized version of ilookup() for file systems where the
 * inode number is not sufficient for unique identification of an inode.
 *
 * Note: @test is called with the inode_hash_lock held, so can't sleep.
 */
struct inode *ilookup5(struct super_block *sb, unsigned long hashval,
		int (*test)(struct inode *, void *), void *data)
{
	struct inode *inode;
	bool isnew;

	might_sleep();

again:
	inode = ilookup5_nowait(sb, hashval, test, data, &isnew);
	if (inode) {
		if (unlikely(isnew))
			wait_on_new_inode(inode);
		if (unlikely(inode_unhashed(inode))) {
			iput(inode);
			goto again;
		}
	}
	return inode;
}
EXPORT_SYMBOL(ilookup5);

/**
 * ilookup - search for an inode in the inode cache
 * @sb:		super block of file system to search
 * @ino:	inode number to search for
 *
 * Search for the inode @ino in the inode cache, and if the inode is in the
 * cache, the inode is returned with an incremented reference count.
 */
struct inode *ilookup(struct super_block *sb, unsigned long ino)
{
	struct hlist_head *head = inode_hashtable + hash(sb, ino);
	struct inode *inode;
	bool isnew;

	might_sleep();

again:
	inode = find_inode_fast(sb, head, ino, false, &isnew);

	if (inode) {
		if (IS_ERR(inode))
			return NULL;
		if (unlikely(isnew))
			wait_on_new_inode(inode);
		if (unlikely(inode_unhashed(inode))) {
			iput(inode);
			goto again;
		}
	}
	return inode;
}
EXPORT_SYMBOL(ilookup);

/**
 * find_inode_nowait - find an inode in the inode cache
 * @sb:		super block of file system to search
 * @hashval:	hash value (usually inode number) to search for
 * @match:	callback used for comparisons between inodes
 * @data:	opaque data pointer to pass to @match
 *
 * Search for the inode specified by @hashval and @data in the inode
 * cache, where the helper function @match will return 0 if the inode
 * does not match, 1 if the inode does match, and -1 if the search
 * should be stopped.  The @match function must be responsible for
 * taking the i_lock spin_lock and checking i_state for an inode being
 * freed or being initialized, and incrementing the reference count
 * before returning 1.  It also must not sleep, since it is called with
 * the inode_hash_lock spinlock held.
 *
 * This is a even more generalized version of ilookup5() when the
 * function must never block --- find_inode() can block in
 * __wait_on_freeing_inode() --- or when the caller can not increment
 * the reference count because the resulting iput() might cause an
 * inode eviction.  The tradeoff is that the @match funtion must be
 * very carefully implemented.
 */
struct inode *find_inode_nowait(struct super_block *sb,
				unsigned long hashval,
				int (*match)(struct inode *, unsigned long,
					     void *),
				void *data)
{
	struct hlist_head *head = inode_hashtable + hash(sb, hashval);
	struct inode *inode, *ret_inode = NULL;
	int mval;

	spin_lock(&inode_hash_lock);
	hlist_for_each_entry(inode, head, i_hash) {
		if (inode->i_sb != sb)
			continue;
		mval = match(inode, hashval, data);
		if (mval == 0)
			continue;
		if (mval == 1)
			ret_inode = inode;
		goto out;
	}
out:
	spin_unlock(&inode_hash_lock);
	return ret_inode;
}
EXPORT_SYMBOL(find_inode_nowait);

/**
 * find_inode_rcu - find an inode in the inode cache
 * @sb:		Super block of file system to search
 * @hashval:	Key to hash
 * @test:	Function to test match on an inode
 * @data:	Data for test function
 *
 * Search for the inode specified by @hashval and @data in the inode cache,
 * where the helper function @test will return 0 if the inode does not match
 * and 1 if it does.  The @test function must be responsible for taking the
 * i_lock spin_lock and checking i_state for an inode being freed or being
 * initialized.
 *
 * If successful, this will return the inode for which the @test function
 * returned 1 and NULL otherwise.
 *
 * The @test function is not permitted to take a ref on any inode presented.
 * It is also not permitted to sleep.
 *
 * The caller must hold the RCU read lock.
 */
struct inode *find_inode_rcu(struct super_block *sb, unsigned long hashval,
			     int (*test)(struct inode *, void *), void *data)
{
	struct hlist_head *head = inode_hashtable + hash(sb, hashval);
	struct inode *inode;

	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
			 "suspicious find_inode_rcu() usage");

	hlist_for_each_entry_rcu(inode, head, i_hash) {
		if (inode->i_sb == sb &&
		    !(inode_state_read_once(inode) & (I_FREEING | I_WILL_FREE)) &&
		    test(inode, data))
			return inode;
	}
	return NULL;
}
EXPORT_SYMBOL(find_inode_rcu);

/**
 * find_inode_by_ino_rcu - Find an inode in the inode cache
 * @sb:		Super block of file system to search
 * @ino:	The inode number to match
 *
 * Search for the inode specified by @hashval and @data in the inode cache,
 * where the helper function @test will return 0 if the inode does not match
 * and 1 if it does.  The @test function must be responsible for taking the
 * i_lock spin_lock and checking i_state for an inode being freed or being
 * initialized.
 *
 * If successful, this will return the inode for which the @test function
 * returned 1 and NULL otherwise.
 *
 * The @test function is not permitted to take a ref on any inode presented.
 * It is also not permitted to sleep.
 *
 * The caller must hold the RCU read lock.
 */
struct inode *find_inode_by_ino_rcu(struct super_block *sb,
				    unsigned long ino)
{
	struct hlist_head *head = inode_hashtable + hash(sb, ino);
	struct inode *inode;

	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
			 "suspicious find_inode_by_ino_rcu() usage");

	hlist_for_each_entry_rcu(inode, head, i_hash) {
		if (inode->i_ino == ino &&
		    inode->i_sb == sb &&
		    !(inode_state_read_once(inode) & (I_FREEING | I_WILL_FREE)))
		    return inode;
	}
	return NULL;
}
EXPORT_SYMBOL(find_inode_by_ino_rcu);

int insert_inode_locked(struct inode *inode)
{
	struct super_block *sb = inode->i_sb;
	ino_t ino = inode->i_ino;
	struct hlist_head *head = inode_hashtable + hash(sb, ino);
	bool isnew;

	might_sleep();

	while (1) {
		struct inode *old = NULL;
		spin_lock(&inode_hash_lock);
repeat:
		hlist_for_each_entry(old, head, i_hash) {
			if (old->i_ino != ino)
				continue;
			if (old->i_sb != sb)
				continue;
			spin_lock(&old->i_lock);
			break;
		}
		if (likely(!old)) {
			spin_lock(&inode->i_lock);
			inode_state_set(inode, I_NEW | I_CREATING);
			hlist_add_head_rcu(&inode->i_hash, head);
			spin_unlock(&inode->i_lock);
			spin_unlock(&inode_hash_lock);
			return 0;
		}
		if (inode_state_read(old) & (I_FREEING | I_WILL_FREE)) {
			__wait_on_freeing_inode(old, true, false);
			old = NULL;
			goto repeat;
		}
		if (unlikely(inode_state_read(old) & I_CREATING)) {
			spin_unlock(&old->i_lock);
			spin_unlock(&inode_hash_lock);
			return -EBUSY;
		}
		__iget(old);
		isnew = !!(inode_state_read(old) & I_NEW);
		spin_unlock(&old->i_lock);
		spin_unlock(&inode_hash_lock);
		if (isnew)
			wait_on_new_inode(old);
		if (unlikely(!inode_unhashed(old))) {
			iput(old);
			return -EBUSY;
		}
		iput(old);
	}
}
EXPORT_SYMBOL(insert_inode_locked);

int insert_inode_locked4(struct inode *inode, unsigned long hashval,
		int (*test)(struct inode *, void *), void *data)
{
	struct inode *old;

	might_sleep();

	inode_state_set_raw(inode, I_CREATING);
	old = inode_insert5(inode, hashval, test, NULL, data);

	if (old != inode) {
		iput(old);
		return -EBUSY;
	}
	return 0;
}
EXPORT_SYMBOL(insert_inode_locked4);


int inode_just_drop(struct inode *inode)
{
	return 1;
}
EXPORT_SYMBOL(inode_just_drop);

/*
 * Called when we're dropping the last reference
 * to an inode.
 *
 * Call the FS "drop_inode()" function, defaulting to
 * the legacy UNIX filesystem behaviour.  If it tells
 * us to evict inode, do so.  Otherwise, retain inode
 * in cache if fs is alive, sync and evict if fs is
 * shutting down.
 */
static void iput_final(struct inode *inode)
{
	struct super_block *sb = inode->i_sb;
	const struct super_operations *op = inode->i_sb->s_op;
	int drop;

	WARN_ON(inode_state_read(inode) & I_NEW);
	VFS_BUG_ON_INODE(atomic_read(&inode->i_count) != 0, inode);

	if (op->drop_inode)
		drop = op->drop_inode(inode);
	else
		drop = inode_generic_drop(inode);

	if (!drop &&
	    !(inode_state_read(inode) & I_DONTCACHE) &&
	    (sb->s_flags & SB_ACTIVE)) {
		__inode_lru_list_add(inode, true);
		spin_unlock(&inode->i_lock);
		return;
	}

	/*
	 * Re-check ->i_count in case the ->drop_inode() hooks played games.
	 * Note we only execute this if the verdict was to drop the inode.
	 */
	VFS_BUG_ON_INODE(atomic_read(&inode->i_count) != 0, inode);

	if (drop) {
		inode_state_set(inode, I_FREEING);
	} else {
		inode_state_set(inode, I_WILL_FREE);
		spin_unlock(&inode->i_lock);

		write_inode_now(inode, 1);

		spin_lock(&inode->i_lock);
		WARN_ON(inode_state_read(inode) & I_NEW);
		inode_state_replace(inode, I_WILL_FREE, I_FREEING);
	}

	inode_lru_list_del(inode);
	spin_unlock(&inode->i_lock);

	evict(inode);
}

/**
 *	iput	- put an inode
 *	@inode: inode to put
 *
 *	Puts an inode, dropping its usage count. If the inode use count hits
 *	zero, the inode is then freed and may also be destroyed.
 *
 *	Consequently, iput() can sleep.
 */
void iput(struct inode *inode)
{
	might_sleep();
	if (unlikely(!inode))
		return;

retry:
	lockdep_assert_not_held(&inode->i_lock);
	VFS_BUG_ON_INODE(inode_state_read_once(inode) & (I_FREEING | I_CLEAR), inode);
	/*
	 * Note this assert is technically racy as if the count is bogusly
	 * equal to one, then two CPUs racing to further drop it can both
	 * conclude it's fine.
	 */
	VFS_BUG_ON_INODE(atomic_read(&inode->i_count) < 1, inode);

	if (atomic_add_unless(&inode->i_count, -1, 1))
		return;

	if (inode->i_nlink && sync_lazytime(inode))
		goto retry;

	spin_lock(&inode->i_lock);
	if (unlikely((inode_state_read(inode) & I_DIRTY_TIME) && inode->i_nlink)) {
		spin_unlock(&inode->i_lock);
		goto retry;
	}

	if (!atomic_dec_and_test(&inode->i_count)) {
		spin_unlock(&inode->i_lock);
		return;
	}

	/*
	 * iput_final() drops ->i_lock, we can't assert on it as the inode may
	 * be deallocated by the time the call returns.
	 */
	iput_final(inode);
}
EXPORT_SYMBOL(iput);

/**
 *	iput_not_last	- put an inode assuming this is not the last reference
 *	@inode: inode to put
 */
void iput_not_last(struct inode *inode)
{
	VFS_BUG_ON_INODE(inode_state_read_once(inode) & (I_FREEING | I_CLEAR), inode);
	VFS_BUG_ON_INODE(atomic_read(&inode->i_count) < 2, inode);

	WARN_ON(atomic_sub_return(1, &inode->i_count) == 0);
}
EXPORT_SYMBOL(iput_not_last);

#ifdef CONFIG_BLOCK
/**
 *	bmap	- find a block number in a file
 *	@inode:  inode owning the block number being requested
 *	@block: pointer containing the block to find
 *
 *	Replaces the value in ``*block`` with the block number on the device holding
 *	corresponding to the requested block number in the file.
 *	That is, asked for block 4 of inode 1 the function will replace the
 *	4 in ``*block``, with disk block relative to the disk start that holds that
 *	block of the file.
 *
 *	Returns -EINVAL in case of error, 0 otherwise. If mapping falls into a
 *	hole, returns 0 and ``*block`` is also set to 0.
 */
int bmap(struct inode *inode, sector_t *block)
{
	if (!inode->i_mapping->a_ops->bmap)
		return -EINVAL;

	*block = inode->i_mapping->a_ops->bmap(inode->i_mapping, *block);
	return 0;
}
EXPORT_SYMBOL(bmap);
#endif

/*
 * With relative atime, only update atime if the previous atime is
 * earlier than or equal to either the ctime or mtime,
 * or if at least a day has passed since the last atime update.
 */
static bool relatime_need_update(struct vfsmount *mnt, struct inode *inode,
			     struct timespec64 now)
{
	struct timespec64 atime, mtime, ctime;

	if (!(mnt->mnt_flags & MNT_RELATIME))
		return true;
	/*
	 * Is mtime younger than or equal to atime? If yes, update atime:
	 */
	atime = inode_get_atime(inode);
	mtime = inode_get_mtime(inode);
	if (timespec64_compare(&mtime, &atime) >= 0)
		return true;
	/*
	 * Is ctime younger than or equal to atime? If yes, update atime:
	 */
	ctime = inode_get_ctime(inode);
	if (timespec64_compare(&ctime, &atime) >= 0)
		return true;

	/*
	 * Is the previous atime value older than a day? If yes,
	 * update atime:
	 */
	if ((long)(now.tv_sec - atime.tv_sec) >= 24*60*60)
		return true;
	/*
	 * Good, we can skip the atime update:
	 */
	return false;
}

static int inode_update_atime(struct inode *inode)
{
	struct timespec64 atime = inode_get_atime(inode);
	struct timespec64 now = current_time(inode);

	if (timespec64_equal(&now, &atime))
		return 0;

	inode_set_atime_to_ts(inode, now);
	return inode_time_dirty_flag(inode);
}

static int inode_update_cmtime(struct inode *inode, unsigned int flags)
{
	struct timespec64 ctime = inode_get_ctime(inode);
	struct timespec64 mtime = inode_get_mtime(inode);
	struct timespec64 now = inode_set_ctime_current(inode);
	unsigned int dirty = 0;
	bool mtime_changed;

	mtime_changed = !timespec64_equal(&now, &mtime);
	if (mtime_changed || !timespec64_equal(&now, &ctime))
		dirty = inode_time_dirty_flag(inode);

	/*
	 * Pure timestamp updates can be recorded in the inode without blocking
	 * by not dirtying the inode.  But when the file system requires
	 * i_version updates, the update of i_version can still block.
	 * Error out if we'd actually have to update i_version or don't support
	 * lazytime.
	 */
	if (IS_I_VERSION(inode)) {
		if (flags & IOCB_NOWAIT) {
			if (!(inode->i_sb->s_flags & SB_LAZYTIME) ||
			    inode_iversion_need_inc(inode))
				return -EAGAIN;
		} else {
			if (inode_maybe_inc_iversion(inode, !!dirty))
				dirty |= I_DIRTY_SYNC;
		}
	}

	if (mtime_changed)
		inode_set_mtime_to_ts(inode, now);
	return dirty;
}

/**
 * inode_update_time - update either atime or c/mtime and i_version on the inode
 * @inode: inode to be updated
 * @type: timestamp to be updated
 * @flags: flags for the update
 *
 * Update either atime or c/mtime and version in a inode if needed for a file
 * access or modification.  It is up to the caller to mark the inode dirty
 * appropriately.
 *
 * Returns the positive I_DIRTY_* flags for __mark_inode_dirty() if the inode
 * needs to be marked dirty, 0 if it did not, or a negative errno if an error
 * happened.
 */
int inode_update_time(struct inode *inode, enum fs_update_time type,
		unsigned int flags)
{
	switch (type) {
	case FS_UPD_ATIME:
		return inode_update_atime(inode);
	case FS_UPD_CMTIME:
		return inode_update_cmtime(inode, flags);
	default:
		WARN_ON_ONCE(1);
		return -EIO;
	}
}
EXPORT_SYMBOL(inode_update_time);

/**
 * generic_update_time - update the timestamps on the inode
 * @inode: inode to be updated
 * @type: timestamp to be updated
 * @flags: flags for the update
 *
 * Returns a negative error value on error, else 0.
 */
int generic_update_time(struct inode *inode, enum fs_update_time type,
		unsigned int flags)
{
	int dirty;

	/*
	 * ->dirty_inode is what could make generic timestamp updates block.
	 * Don't support non-blocking timestamp updates here if it is set.
	 * File systems that implement ->dirty_inode but want to support
	 * non-blocking timestamp updates should call inode_update_time
	 * directly.
	 */
	if ((flags & IOCB_NOWAIT) && inode->i_sb->s_op->dirty_inode)
		return -EAGAIN;

	dirty = inode_update_time(inode, type, flags);
	if (dirty <= 0)
		return dirty;
	__mark_inode_dirty(inode, dirty);
	return 0;
}
EXPORT_SYMBOL(generic_update_time);

/**
 *	atime_needs_update	-	update the access time
 *	@path: the &struct path to update
 *	@inode: inode to update
 *
 *	Update the accessed time on an inode and mark it for writeback.
 *	This function automatically handles read only file systems and media,
 *	as well as the "noatime" flag and inode specific "noatime" markers.
 */
bool atime_needs_update(const struct path *path, struct inode *inode)
{
	struct vfsmount *mnt = path->mnt;
	struct timespec64 now, atime;

	if (inode->i_flags & S_NOATIME)
		return false;

	/* Atime updates will likely cause i_uid and i_gid to be written
	 * back improprely if their true value is unknown to the vfs.
	 */
	if (HAS_UNMAPPED_ID(mnt_idmap(mnt), inode))
		return false;

	if (IS_NOATIME(inode))
		return false;
	if ((inode->i_sb->s_flags & SB_NODIRATIME) && S_ISDIR(inode->i_mode))
		return false;

	if (mnt->mnt_flags & MNT_NOATIME)
		return false;
	if ((mnt->mnt_flags & MNT_NODIRATIME) && S_ISDIR(inode->i_mode))
		return false;

	now = current_time(inode);

	if (!relatime_need_update(mnt, inode, now))
		return false;

	atime = inode_get_atime(inode);
	if (timespec64_equal(&atime, &now))
		return false;

	return true;
}

void touch_atime(const struct path *path)
{
	struct vfsmount *mnt = path->mnt;
	struct inode *inode = d_inode(path->dentry);

	if (!atime_needs_update(path, inode))
		return;

	if (!sb_start_write_trylock(inode->i_sb))
		return;

	if (mnt_get_write_access(mnt) != 0)
		goto skip_update;
	/*
	 * File systems can error out when updating inodes if they need to
	 * allocate new space to modify an inode (such is the case for
	 * Btrfs), but since we touch atime while walking down the path we
	 * really don't care if we failed to update the atime of the file,
	 * so just ignore the return value.
	 * We may also fail on filesystems that have the ability to make parts
	 * of the fs read only, e.g. subvolumes in Btrfs.
	 */
	if (inode->i_op->update_time)
		inode->i_op->update_time(inode, FS_UPD_ATIME, 0);
	else
		generic_update_time(inode, FS_UPD_ATIME, 0);
	mnt_put_write_access(mnt);
skip_update:
	sb_end_write(inode->i_sb);
}
EXPORT_SYMBOL(touch_atime);

/*
 * Return mask of changes for notify_change() that need to be done as a
 * response to write or truncate. Return 0 if nothing has to be changed.
 * Negative value on error (change should be denied).
 */
int dentry_needs_remove_privs(struct mnt_idmap *idmap,
			      struct dentry *dentry)
{
	struct inode *inode = d_inode(dentry);
	int mask = 0;
	int ret;

	if (IS_NOSEC(inode))
		return 0;

	mask = setattr_should_drop_suidgid(idmap, inode);
	ret = security_inode_need_killpriv(dentry);
	if (ret < 0)
		return ret;
	if (ret)
		mask |= ATTR_KILL_PRIV;
	return mask;
}

static int __remove_privs(struct mnt_idmap *idmap,
			  struct dentry *dentry, int kill)
{
	struct iattr newattrs;

	newattrs.ia_valid = ATTR_FORCE | kill;
	/*
	 * Note we call this on write, so notify_change will not
	 * encounter any conflicting delegations:
	 */
	return notify_change(idmap, dentry, &newattrs, NULL);
}

static int file_remove_privs_flags(struct file *file, unsigned int flags)
{
	struct dentry *dentry = file_dentry(file);
	struct inode *inode = file_inode(file);
	int error = 0;
	int kill;

	if (IS_NOSEC(inode) || !S_ISREG(inode->i_mode))
		return 0;

	kill = dentry_needs_remove_privs(file_mnt_idmap(file), dentry);
	if (kill < 0)
		return kill;

	if (kill) {
		if (flags & IOCB_NOWAIT)
			return -EAGAIN;

		error = __remove_privs(file_mnt_idmap(file), dentry, kill);
	}

	if (!error)
		inode_has_no_xattr(inode);
	return error;
}

/**
 * file_remove_privs - remove special file privileges (suid, capabilities)
 * @file: file to remove privileges from
 *
 * When file is modified by a write or truncation ensure that special
 * file privileges are removed.
 *
 * Return: 0 on success, negative errno on failure.
 */
int file_remove_privs(struct file *file)
{
	return file_remove_privs_flags(file, 0);
}
EXPORT_SYMBOL(file_remove_privs);

/**
 * current_time - Return FS time (possibly fine-grained)
 * @inode: inode.
 *
 * Return the current time truncated to the time granularity supported by
 * the fs, as suitable for a ctime/mtime change. If the ctime is flagged
 * as having been QUERIED, get a fine-grained timestamp, but don't update
 * the floor.
 *
 * For a multigrain inode, this is effectively an estimate of the timestamp
 * that a file would receive. An actual update must go through
 * inode_set_ctime_current().
 */
struct timespec64 current_time(struct inode *inode)
{
	struct timespec64 now;
	u32 cns;

	ktime_get_coarse_real_ts64_mg(&now);

	if (!is_mgtime(inode))
		goto out;

	/* If nothing has queried it, then coarse time is fine */
	cns = smp_load_acquire(&inode->i_ctime_nsec);
	if (cns & I_CTIME_QUERIED) {
		/*
		 * If there is no apparent change, then get a fine-grained
		 * timestamp.
		 */
		if (now.tv_nsec == (cns & ~I_CTIME_QUERIED))
			ktime_get_real_ts64(&now);
	}
out:
	return timestamp_truncate(now, inode);
}
EXPORT_SYMBOL(current_time);

static inline bool need_cmtime_update(struct inode *inode)
{
	struct timespec64 now = current_time(inode), ts;

	ts = inode_get_mtime(inode);
	if (!timespec64_equal(&ts, &now))
		return true;
	ts = inode_get_ctime(inode);
	if (!timespec64_equal(&ts, &now))
		return true;
	return IS_I_VERSION(inode) && inode_iversion_need_inc(inode);
}

static int file_update_time_flags(struct file *file, unsigned int flags)
{
	struct inode *inode = file_inode(file);
	int ret;

	/* First try to exhaust all avenues to not sync */
	if (IS_NOCMTIME(inode))
		return 0;
	if (unlikely(file->f_mode & FMODE_NOCMTIME))
		return 0;
	if (!need_cmtime_update(inode))
		return 0;

	flags &= IOCB_NOWAIT;
	if (mnt_get_write_access_file(file))
		return 0;
	if (inode->i_op->update_time)
		ret = inode->i_op->update_time(inode, FS_UPD_CMTIME, flags);
	else
		ret = generic_update_time(inode, FS_UPD_CMTIME, flags);
	mnt_put_write_access_file(file);
	return ret;
}

/**
 * file_update_time - update mtime and ctime time
 * @file: file accessed
 *
 * Update the mtime and ctime members of an inode and mark the inode for
 * writeback. Note that this function is meant exclusively for usage in
 * the file write path of filesystems, and filesystems may choose to
 * explicitly ignore updates via this function with the _NOCMTIME inode
 * flag, e.g. for network filesystem where these imestamps are handled
 * by the server. This can return an error for file systems who need to
 * allocate space in order to update an inode.
 *
 * Return: 0 on success, negative errno on failure.
 */
int file_update_time(struct file *file)
{
	return file_update_time_flags(file, 0);
}
EXPORT_SYMBOL(file_update_time);

/**
 * file_modified_flags - handle mandated vfs changes when modifying a file
 * @file: file that was modified
 * @flags: kiocb flags
 *
 * When file has been modified ensure that special
 * file privileges are removed and time settings are updated.
 *
 * If IOCB_NOWAIT is set, special file privileges will not be removed and
 * time settings will not be updated. It will return -EAGAIN.
 *
 * Context: Caller must hold the file's inode lock.
 *
 * Return: 0 on success, negative errno on failure.
 */
static int file_modified_flags(struct file *file, int flags)
{
	int ret;

	/*
	 * Clear the security bits if the process is not being run by root.
	 * This keeps people from modifying setuid and setgid binaries.
	 */
	ret = file_remove_privs_flags(file, flags);
	if (ret)
		return ret;
	return file_update_time_flags(file, flags);
}

/**
 * file_modified - handle mandated vfs changes when modifying a file
 * @file: file that was modified
 *
 * When file has been modified ensure that special
 * file privileges are removed and time settings are updated.
 *
 * Context: Caller must hold the file's inode lock.
 *
 * Return: 0 on success, negative errno on failure.
 */
int file_modified(struct file *file)
{
	return file_modified_flags(file, 0);
}
EXPORT_SYMBOL(file_modified);

/**
 * kiocb_modified - handle mandated vfs changes when modifying a file
 * @iocb: iocb that was modified
 *
 * When file has been modified ensure that special
 * file privileges are removed and time settings are updated.
 *
 * Context: Caller must hold the file's inode lock.
 *
 * Return: 0 on success, negative errno on failure.
 */
int kiocb_modified(struct kiocb *iocb)
{
	return file_modified_flags(iocb->ki_filp, iocb->ki_flags);
}
EXPORT_SYMBOL_GPL(kiocb_modified);

int inode_needs_sync(struct inode *inode)
{
	if (IS_SYNC(inode))
		return 1;
	if (S_ISDIR(inode->i_mode) && IS_DIRSYNC(inode))
		return 1;
	return 0;
}
EXPORT_SYMBOL(inode_needs_sync);

/*
 * If we try to find an inode in the inode hash while it is being
 * deleted, we have to wait until the filesystem completes its
 * deletion before reporting that it isn't found.  This function waits
 * until the deletion _might_ have completed.  Callers are responsible
 * to recheck inode state.
 *
 * It doesn't matter if I_NEW is not set initially, a call to
 * wake_up_bit(&inode->i_state, __I_NEW) after removing from the hash list
 * will DTRT.
 */
static void __wait_on_freeing_inode(struct inode *inode, bool hash_locked, bool rcu_locked)
{
	struct wait_bit_queue_entry wqe;
	struct wait_queue_head *wq_head;

	VFS_BUG_ON(!hash_locked && !rcu_locked);

	/*
	 * Handle racing against evict(), see that routine for more details.
	 */
	if (unlikely(inode_unhashed(inode))) {
		WARN_ON(hash_locked);
		spin_unlock(&inode->i_lock);
		return;
	}

	wq_head = inode_bit_waitqueue(&wqe, inode, __I_NEW);
	prepare_to_wait_event(wq_head, &wqe.wq_entry, TASK_UNINTERRUPTIBLE);
	spin_unlock(&inode->i_lock);
	if (rcu_locked)
		rcu_read_unlock();
	if (hash_locked)
		spin_unlock(&inode_hash_lock);
	schedule();
	finish_wait(wq_head, &wqe.wq_entry);
	if (hash_locked)
		spin_lock(&inode_hash_lock);
	if (rcu_locked)
		rcu_read_lock();
}

static __initdata unsigned long ihash_entries;
static int __init set_ihash_entries(char *str)
{
	return kstrtoul(str, 0, &ihash_entries) == 0;
}
__setup("ihash_entries=", set_ihash_entries);

/*
 * Initialize the waitqueues and inode hash table.
 */
void __init inode_init_early(void)
{
	/* If hashes are distributed across NUMA nodes, defer
	 * hash allocation until vmalloc space is available.
	 */
	if (hashdist)
		return;

	inode_hashtable =
		alloc_large_system_hash("Inode-cache",
					sizeof(struct hlist_head),
					ihash_entries,
					14,
					HASH_EARLY | HASH_ZERO,
					&i_hash_shift,
					&i_hash_mask,
					0,
					0);
}

void __init inode_init(void)
{
	/* inode slab cache */
	inode_cachep = kmem_cache_create("inode_cache",
					 sizeof(struct inode),
					 0,
					 (SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|
					 SLAB_ACCOUNT),
					 init_once);

	/* Hash may have been set up in inode_init_early */
	if (!hashdist)
		return;

	inode_hashtable =
		alloc_large_system_hash("Inode-cache",
					sizeof(struct hlist_head),
					ihash_entries,
					14,
					HASH_ZERO,
					&i_hash_shift,
					&i_hash_mask,
					0,
					0);
}

void init_special_inode(struct inode *inode, umode_t mode, dev_t rdev)
{
	inode->i_mode = mode;
	switch (inode->i_mode & S_IFMT) {
	case S_IFCHR:
		inode->i_fop = &def_chr_fops;
		inode->i_rdev = rdev;
		break;
	case S_IFBLK:
		if (IS_ENABLED(CONFIG_BLOCK))
			inode->i_fop = &def_blk_fops;
		inode->i_rdev = rdev;
		break;
	case S_IFIFO:
		inode->i_fop = &pipefifo_fops;
		break;
	case S_IFSOCK:
		/* leave it no_open_fops */
		break;
	default:
		printk(KERN_DEBUG "init_special_inode: bogus i_mode (%o) for"
				  " inode %s:%lu\n", mode, inode->i_sb->s_id,
				  inode->i_ino);
		break;
	}
}
EXPORT_SYMBOL(init_special_inode);

/**
 * inode_init_owner - Init uid,gid,mode for new inode according to posix standards
 * @idmap: idmap of the mount the inode was created from
 * @inode: New inode
 * @dir: Directory inode
 * @mode: mode of the new inode
 *
 * If the inode has been created through an idmapped mount the idmap of
 * the vfsmount must be passed through @idmap. This function will then take
 * care to map the inode according to @idmap before checking permissions
 * and initializing i_uid and i_gid. On non-idmapped mounts or if permission
 * checking is to be performed on the raw inode simply pass @nop_mnt_idmap.
 */
void inode_init_owner(struct mnt_idmap *idmap, struct inode *inode,
		      const struct inode *dir, umode_t mode)
{
	inode_fsuid_set(inode, idmap);
	if (dir && dir->i_mode & S_ISGID) {
		inode->i_gid = dir->i_gid;

		/* Directories are special, and always inherit S_ISGID */
		if (S_ISDIR(mode))
			mode |= S_ISGID;
	} else
		inode_fsgid_set(inode, idmap);
	inode->i_mode = mode;
}
EXPORT_SYMBOL(inode_init_owner);

/**
 * inode_owner_or_capable - check current task permissions to inode
 * @idmap: idmap of the mount the inode was found from
 * @inode: inode being checked
 *
 * Return true if current either has CAP_FOWNER in a namespace with the
 * inode owner uid mapped, or owns the file.
 *
 * If the inode has been found through an idmapped mount the idmap of
 * the vfsmount must be passed through @idmap. This function will then take
 * care to map the inode according to @idmap before checking permissions.
 * On non-idmapped mounts or if permission checking is to be performed on the
 * raw inode simply pass @nop_mnt_idmap.
 */
bool inode_owner_or_capable(struct mnt_idmap *idmap,
			    const struct inode *inode)
{
	vfsuid_t vfsuid;
	struct user_namespace *ns;

	vfsuid = i_uid_into_vfsuid(idmap, inode);
	if (vfsuid_eq_kuid(vfsuid, current_fsuid()))
		return true;

	ns = current_user_ns();
	if (vfsuid_has_mapping(ns, vfsuid) && ns_capable(ns, CAP_FOWNER))
		return true;
	return false;
}
EXPORT_SYMBOL(inode_owner_or_capable);

/*
 * Direct i/o helper functions
 */
bool inode_dio_finished(const struct inode *inode)
{
	return atomic_read(&inode->i_dio_count) == 0;
}
EXPORT_SYMBOL(inode_dio_finished);

/**
 * inode_dio_wait - wait for outstanding DIO requests to finish
 * @inode: inode to wait for
 *
 * Waits for all pending direct I/O requests to finish so that we can
 * proceed with a truncate or equivalent operation.
 *
 * Must be called under a lock that serializes taking new references
 * to i_dio_count, usually by inode->i_rwsem.
 */
void inode_dio_wait(struct inode *inode)
{
	wait_var_event(&inode->i_dio_count, inode_dio_finished(inode));
}
EXPORT_SYMBOL(inode_dio_wait);

void inode_dio_wait_interruptible(struct inode *inode)
{
	wait_var_event_interruptible(&inode->i_dio_count,
				     inode_dio_finished(inode));
}
EXPORT_SYMBOL(inode_dio_wait_interruptible);

/*
 * inode_set_flags - atomically set some inode flags
 *
 * Note: the caller should be holding i_rwsem exclusively, or else be sure that
 * they have exclusive access to the inode structure (i.e., while the
 * inode is being instantiated).  The reason for the cmpxchg() loop
 * --- which wouldn't be necessary if all code paths which modify
 * i_flags actually followed this rule, is that there is at least one
 * code path which doesn't today so we use cmpxchg() out of an abundance
 * of caution.
 *
 * In the long run, i_rwsem is overkill, and we should probably look
 * at using the i_lock spinlock to protect i_flags, and then make sure
 * it is so documented in include/linux/fs.h and that all code follows
 * the locking convention!!
 */
void inode_set_flags(struct inode *inode, unsigned int flags,
		     unsigned int mask)
{
	WARN_ON_ONCE(flags & ~mask);
	set_mask_bits(&inode->i_flags, mask, flags);
}
EXPORT_SYMBOL(inode_set_flags);

void inode_nohighmem(struct inode *inode)
{
	mapping_set_gfp_mask(inode->i_mapping, GFP_USER);
}
EXPORT_SYMBOL(inode_nohighmem);

struct timespec64 inode_set_ctime_to_ts(struct inode *inode, struct timespec64 ts)
{
	trace_inode_set_ctime_to_ts(inode, &ts);
	set_normalized_timespec64(&ts, ts.tv_sec, ts.tv_nsec);
	inode->i_ctime_sec = ts.tv_sec;
	inode->i_ctime_nsec = ts.tv_nsec;
	return ts;
}
EXPORT_SYMBOL(inode_set_ctime_to_ts);

/**
 * timestamp_truncate - Truncate timespec to a granularity
 * @t: Timespec
 * @inode: inode being updated
 *
 * Truncate a timespec to the granularity supported by the fs
 * containing the inode. Always rounds down. gran must
 * not be 0 nor greater than a second (NSEC_PER_SEC, or 10^9 ns).
 */
struct timespec64 timestamp_truncate(struct timespec64 t, struct inode *inode)
{
	struct super_block *sb = inode->i_sb;
	unsigned int gran = sb->s_time_gran;

	t.tv_sec = clamp(t.tv_sec, sb->s_time_min, sb->s_time_max);
	if (unlikely(t.tv_sec == sb->s_time_max || t.tv_sec == sb->s_time_min))
		t.tv_nsec = 0;

	/* Avoid division in the common cases 1 ns and 1 s. */
	if (gran == 1)
		; /* nothing */
	else if (gran == NSEC_PER_SEC)
		t.tv_nsec = 0;
	else if (gran > 1 && gran < NSEC_PER_SEC)
		t.tv_nsec -= t.tv_nsec % gran;
	else
		WARN(1, "invalid file time granularity: %u", gran);
	return t;
}
EXPORT_SYMBOL(timestamp_truncate);

/**
 * inode_set_ctime_current - set the ctime to current_time
 * @inode: inode
 *
 * Set the inode's ctime to the current value for the inode. Returns the
 * current value that was assigned. If this is not a multigrain inode, then we
 * set it to the later of the coarse time and floor value.
 *
 * If it is multigrain, then we first see if the coarse-grained timestamp is
 * distinct from what is already there. If so, then use that. Otherwise, get a
 * fine-grained timestamp.
 *
 * After that, try to swap the new value into i_ctime_nsec. Accept the
 * resulting ctime, regardless of the outcome of the swap. If it has
 * already been replaced, then that timestamp is later than the earlier
 * unacceptable one, and is thus acceptable.
 */
struct timespec64 inode_set_ctime_current(struct inode *inode)
{
	struct timespec64 now;
	u32 cns, cur;

	ktime_get_coarse_real_ts64_mg(&now);
	now = timestamp_truncate(now, inode);

	/* Just return that if this is not a multigrain fs */
	if (!is_mgtime(inode)) {
		inode_set_ctime_to_ts(inode, now);
		goto out;
	}

	/*
	 * A fine-grained time is only needed if someone has queried
	 * for timestamps, and the current coarse grained time isn't
	 * later than what's already there.
	 */
	cns = smp_load_acquire(&inode->i_ctime_nsec);
	if (cns & I_CTIME_QUERIED) {
		struct timespec64 ctime = { .tv_sec = inode->i_ctime_sec,
					    .tv_nsec = cns & ~I_CTIME_QUERIED };

		if (timespec64_compare(&now, &ctime) <= 0) {
			ktime_get_real_ts64_mg(&now);
			now = timestamp_truncate(now, inode);
			mgtime_counter_inc(mg_fine_stamps);
		}
	}
	mgtime_counter_inc(mg_ctime_updates);

	/* No need to cmpxchg if it's exactly the same */
	if (cns == now.tv_nsec && inode->i_ctime_sec == now.tv_sec) {
		trace_ctime_xchg_skip(inode, &now);
		goto out;
	}
	cur = cns;
retry:
	/* Try to swap the nsec value into place. */
	if (try_cmpxchg(&inode->i_ctime_nsec, &cur, now.tv_nsec)) {
		/* If swap occurred, then we're (mostly) done */
		inode->i_ctime_sec = now.tv_sec;
		trace_ctime_ns_xchg(inode, cns, now.tv_nsec, cur);
		mgtime_counter_inc(mg_ctime_swaps);
	} else {
		/*
		 * Was the change due to someone marking the old ctime QUERIED?
		 * If so then retry the swap. This can only happen once since
		 * the only way to clear I_CTIME_QUERIED is to stamp the inode
		 * with a new ctime.
		 */
		if (!(cns & I_CTIME_QUERIED) && (cns | I_CTIME_QUERIED) == cur) {
			cns = cur;
			goto retry;
		}
		/* Otherwise, keep the existing ctime */
		now.tv_sec = inode->i_ctime_sec;
		now.tv_nsec = cur & ~I_CTIME_QUERIED;
	}
out:
	return now;
}
EXPORT_SYMBOL(inode_set_ctime_current);

/**
 * inode_set_ctime_deleg - try to update the ctime on a delegated inode
 * @inode: inode to update
 * @update: timespec64 to set the ctime
 *
 * Attempt to atomically update the ctime on behalf of a delegation holder.
 *
 * The nfs server can call back the holder of a delegation to get updated
 * inode attributes, including the mtime. When updating the mtime, update
 * the ctime to a value at least equal to that.
 *
 * This can race with concurrent updates to the inode, in which
 * case the update is skipped.
 *
 * Note that this works even when multigrain timestamps are not enabled,
 * so it is used in either case.
 */
struct timespec64 inode_set_ctime_deleg(struct inode *inode, struct timespec64 update)
{
	struct timespec64 now, cur_ts;
	u32 cur, old;

	/* pairs with try_cmpxchg below */
	cur = smp_load_acquire(&inode->i_ctime_nsec);
	cur_ts.tv_nsec = cur & ~I_CTIME_QUERIED;
	cur_ts.tv_sec = inode->i_ctime_sec;

	/* If the update is older than the existing value, skip it. */
	if (timespec64_compare(&update, &cur_ts) <= 0)
		return cur_ts;

	ktime_get_coarse_real_ts64_mg(&now);

	/* Clamp the update to "now" if it's in the future */
	if (timespec64_compare(&update, &now) > 0)
		update = now;

	update = timestamp_truncate(update, inode);

	/* No need to update if the values are already the same */
	if (timespec64_equal(&update, &cur_ts))
		return cur_ts;

	/*
	 * Try to swap the nsec value into place. If it fails, that means
	 * it raced with an update due to a write or similar activity. That
	 * stamp takes precedence, so just skip the update.
	 */
retry:
	old = cur;
	if (try_cmpxchg(&inode->i_ctime_nsec, &cur, update.tv_nsec)) {
		inode->i_ctime_sec = update.tv_sec;
		mgtime_counter_inc(mg_ctime_swaps);
		return update;
	}

	/*
	 * Was the change due to another task marking the old ctime QUERIED?
	 *
	 * If so, then retry the swap. This can only happen once since
	 * the only way to clear I_CTIME_QUERIED is to stamp the inode
	 * with a new ctime.
	 */
	if (!(old & I_CTIME_QUERIED) && (cur == (old | I_CTIME_QUERIED)))
		goto retry;

	/* Otherwise, it was a new timestamp. */
	cur_ts.tv_sec = inode->i_ctime_sec;
	cur_ts.tv_nsec = cur & ~I_CTIME_QUERIED;
	return cur_ts;
}
EXPORT_SYMBOL(inode_set_ctime_deleg);

/**
 * in_group_or_capable - check whether caller is CAP_FSETID privileged
 * @idmap:	idmap of the mount @inode was found from
 * @inode:	inode to check
 * @vfsgid:	the new/current vfsgid of @inode
 *
 * Check whether @vfsgid is in the caller's group list or if the caller is
 * privileged with CAP_FSETID over @inode. This can be used to determine
 * whether the setgid bit can be kept or must be dropped.
 *
 * Return: true if the caller is sufficiently privileged, false if not.
 */
bool in_group_or_capable(struct mnt_idmap *idmap,
			 const struct inode *inode, vfsgid_t vfsgid)
{
	if (vfsgid_in_group_p(vfsgid))
		return true;
	if (capable_wrt_inode_uidgid(idmap, inode, CAP_FSETID))
		return true;
	return false;
}
EXPORT_SYMBOL(in_group_or_capable);

/**
 * mode_strip_sgid - handle the sgid bit for non-directories
 * @idmap: idmap of the mount the inode was created from
 * @dir: parent directory inode
 * @mode: mode of the file to be created in @dir
 *
 * If the @mode of the new file has both the S_ISGID and S_IXGRP bit
 * raised and @dir has the S_ISGID bit raised ensure that the caller is
 * either in the group of the parent directory or they have CAP_FSETID
 * in their user namespace and are privileged over the parent directory.
 * In all other cases, strip the S_ISGID bit from @mode.
 *
 * Return: the new mode to use for the file
 */
umode_t mode_strip_sgid(struct mnt_idmap *idmap,
			const struct inode *dir, umode_t mode)
{
	if ((mode & (S_ISGID | S_IXGRP)) != (S_ISGID | S_IXGRP))
		return mode;
	if (S_ISDIR(mode) || !dir || !(dir->i_mode & S_ISGID))
		return mode;
	if (in_group_or_capable(idmap, dir, i_gid_into_vfsgid(idmap, dir)))
		return mode;
	return mode & ~S_ISGID;
}
EXPORT_SYMBOL(mode_strip_sgid);

#ifdef CONFIG_DEBUG_VFS
/**
 * dump_inode - dump an inode.
 * @inode: inode to dump
 * @reason: reason for dumping
 *
 * If inode is an invalid pointer, we don't want to crash accessing it,
 * so probe everything depending on it carefully with get_kernel_nofault().
 */
void dump_inode(struct inode *inode, const char *reason)
{
	struct super_block *sb;
	struct file_system_type *s_type;
	const char *fs_name_ptr;
	char fs_name[32] = {};
	umode_t mode;
	unsigned short opflags;
	unsigned int flags;
	unsigned int state;
	int count;

	if (get_kernel_nofault(sb, &inode->i_sb) ||
	    get_kernel_nofault(mode, &inode->i_mode) ||
	    get_kernel_nofault(opflags, &inode->i_opflags) ||
	    get_kernel_nofault(flags, &inode->i_flags)) {
		pr_warn("%s: unreadable inode:%px\n", reason, inode);
		return;
	}

	state = inode_state_read_once(inode);
	count = atomic_read(&inode->i_count);

	if (!sb ||
	    get_kernel_nofault(s_type, &sb->s_type) || !s_type ||
	    get_kernel_nofault(fs_name_ptr, &s_type->name) || !fs_name_ptr ||
	    strncpy_from_kernel_nofault(fs_name, fs_name_ptr, sizeof(fs_name) - 1) < 0)
		strscpy(fs_name, "<unknown, sb unreadable>");

	pr_warn("%s: inode:%px fs:%s mode:%ho opflags:%#x flags:%#x state:%#x count:%d\n",
		reason, inode, fs_name, mode, opflags, flags, state, count);
}
EXPORT_SYMBOL(dump_inode);
#endif
]

gemini-3-flash-preview

							input: 55631
							output: 21
							thoughts: 410
						

**Analyzing `inode_set_ctime_current` and `is_mgtime`**

Okay, let's break this down. My initial observation is focused on the `inode_set_ctime_current` function. The code snippet starts at line 2832, and immediately grabs the current time using `ktime_get_coarse_real_ts64_mg`.  Then the timestamp gets truncated based on the inode's configuration. The crucial part for me is the conditional: `if (!is_mgtime(inode))`.  If this condition holds (meaning `is_mgtime` returns false), the code directly sets the ctime using `inode_set_ctime_to_ts`.

I was a bit concerned about the line numbers at first because the numbers in the provided code snippet didn't immediately align with the numbers I'm looking at in my file. But the report says `inode_set_ctime_current+0x256/0x960 fs/inode.c:2842`. That actually matches the `inode_set_ctime_to_ts` line perfectly. So, the report's line numbers are correct after all.

Now, the question is why `is_mgtime(inode)` is returning false for this shmem inode. I recall seeing `FS_MGTIME` associated with `shmem_fs_type` earlier. That suggests that shmem *should* be multigrain, which means `is_mgtime` *should* be true.  I need to double-check `shmem_fs_type` in `mm/shmem.c` to be certain. There must be something else at play here.

map[File:mm/shmem.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0
/*
 * Resizable virtual memory filesystem for Linux.
 *
 * Copyright (C) 2000 Linus Torvalds.
 *		 2000 Transmeta Corp.
 *		 2000-2001 Christoph Rohland
 *		 2000-2001 SAP AG
 *		 2002 Red Hat Inc.
 * Copyright (C) 2002-2011 Hugh Dickins.
 * Copyright (C) 2011 Google Inc.
 * Copyright (C) 2002-2005 VERITAS Software Corporation.
 * Copyright (C) 2004 Andi Kleen, SuSE Labs
 *
 * Extended attribute support for tmpfs:
 * Copyright (c) 2004, Luke Kenneth Casson Leighton <lkcl@lkcl.net>
 * Copyright (c) 2004 Red Hat, Inc., James Morris <jmorris@redhat.com>
 *
 * tiny-shmem:
 * Copyright (c) 2004, 2008 Matt Mackall <mpm@selenic.com>
 */

#include <linux/fs.h>
#include <linux/init.h>
#include <linux/vfs.h>
#include <linux/mount.h>
#include <linux/ramfs.h>
#include <linux/pagemap.h>
#include <linux/file.h>
#include <linux/fileattr.h>
#include <linux/filelock.h>
#include <linux/mm.h>
#include <linux/random.h>
#include <linux/sched/signal.h>
#include <linux/export.h>
#include <linux/shmem_fs.h>
#include <linux/swap.h>
#include <linux/uio.h>
#include <linux/hugetlb.h>
#include <linux/fs_parser.h>
#include <linux/swapfile.h>
#include <linux/iversion.h>
#include <linux/unicode.h>
#include "swap.h"

static struct vfsmount *shm_mnt __ro_after_init;

#ifdef CONFIG_SHMEM
/*
 * This virtual memory filesystem is heavily based on the ramfs. It
 * extends ramfs by the ability to use swap and honor resource limits
 * which makes it a completely usable filesystem.
 */

#include <linux/xattr.h>
#include <linux/exportfs.h>
#include <linux/posix_acl.h>
#include <linux/posix_acl_xattr.h>
#include <linux/mman.h>
#include <linux/string.h>
#include <linux/slab.h>
#include <linux/backing-dev.h>
#include <linux/writeback.h>
#include <linux/pagevec.h>
#include <linux/percpu_counter.h>
#include <linux/falloc.h>
#include <linux/splice.h>
#include <linux/security.h>
#include <linux/leafops.h>
#include <linux/mempolicy.h>
#include <linux/namei.h>
#include <linux/ctype.h>
#include <linux/migrate.h>
#include <linux/highmem.h>
#include <linux/seq_file.h>
#include <linux/magic.h>
#include <linux/syscalls.h>
#include <linux/fcntl.h>
#include <uapi/linux/memfd.h>
#include <linux/rmap.h>
#include <linux/uuid.h>
#include <linux/quotaops.h>
#include <linux/rcupdate_wait.h>

#include <linux/uaccess.h>

#include "internal.h"

#define VM_ACCT(size)    (PAGE_ALIGN(size) >> PAGE_SHIFT)

/* Pretend that each entry is of this size in directory's i_size */
#define BOGO_DIRENT_SIZE 20

/* Pretend that one inode + its dentry occupy this much memory */
#define BOGO_INODE_SIZE 1024

/* Symlink up to this size is kmalloc'ed instead of using a swappable page */
#define SHORT_SYMLINK_LEN 128

/*
 * shmem_fallocate communicates with shmem_fault or shmem_writeout via
 * inode->i_private (with i_rwsem making sure that it has only one user at
 * a time): we would prefer not to enlarge the shmem inode just for that.
 */
struct shmem_falloc {
	wait_queue_head_t *waitq; /* faults into hole wait for punch to end */
	pgoff_t start;		/* start of range currently being fallocated */
	pgoff_t next;		/* the next page offset to be fallocated */
	pgoff_t nr_falloced;	/* how many new pages have been fallocated */
	pgoff_t nr_unswapped;	/* how often writeout refused to swap out */
};

struct shmem_options {
	unsigned long long blocks;
	unsigned long long inodes;
	struct mempolicy *mpol;
	kuid_t uid;
	kgid_t gid;
	umode_t mode;
	bool full_inums;
	int huge;
	int seen;
	bool noswap;
	unsigned short quota_types;
	struct shmem_quota_limits qlimits;
#if IS_ENABLED(CONFIG_UNICODE)
	struct unicode_map *encoding;
	bool strict_encoding;
#endif
#define SHMEM_SEEN_BLOCKS 1
#define SHMEM_SEEN_INODES 2
#define SHMEM_SEEN_HUGE 4
#define SHMEM_SEEN_INUMS 8
#define SHMEM_SEEN_QUOTA 16
};

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
static unsigned long huge_shmem_orders_always __read_mostly;
static unsigned long huge_shmem_orders_madvise __read_mostly;
static unsigned long huge_shmem_orders_inherit __read_mostly;
static unsigned long huge_shmem_orders_within_size __read_mostly;
static bool shmem_orders_configured __initdata;
#endif

#ifdef CONFIG_TMPFS
static unsigned long shmem_default_max_blocks(void)
{
	return totalram_pages() / 2;
}

static unsigned long shmem_default_max_inodes(void)
{
	unsigned long nr_pages = totalram_pages();

	return min3(nr_pages - totalhigh_pages(), nr_pages / 2,
			ULONG_MAX / BOGO_INODE_SIZE);
}
#endif

static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
			struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
			struct vm_area_struct *vma, vm_fault_t *fault_type);

static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
{
	return sb->s_fs_info;
}

/*
 * shmem_file_setup pre-accounts the whole fixed size of a VM object,
 * for shared memory and for shared anonymous (/dev/zero) mappings
 * (unless MAP_NORESERVE and sysctl_overcommit_memory <= 1),
 * consistent with the pre-accounting of private mappings ...
 */
static inline int shmem_acct_size(unsigned long flags, loff_t size)
{
	return (flags & SHMEM_F_NORESERVE) ?
		0 : security_vm_enough_memory_mm(current->mm, VM_ACCT(size));
}

static inline void shmem_unacct_size(unsigned long flags, loff_t size)
{
	if (!(flags & SHMEM_F_NORESERVE))
		vm_unacct_memory(VM_ACCT(size));
}

static inline int shmem_reacct_size(unsigned long flags,
		loff_t oldsize, loff_t newsize)
{
	if (!(flags & SHMEM_F_NORESERVE)) {
		if (VM_ACCT(newsize) > VM_ACCT(oldsize))
			return security_vm_enough_memory_mm(current->mm,
					VM_ACCT(newsize) - VM_ACCT(oldsize));
		else if (VM_ACCT(newsize) < VM_ACCT(oldsize))
			vm_unacct_memory(VM_ACCT(oldsize) - VM_ACCT(newsize));
	}
	return 0;
}

/*
 * ... whereas tmpfs objects are accounted incrementally as
 * pages are allocated, in order to allow large sparse files.
 * shmem_get_folio reports shmem_acct_blocks failure as -ENOSPC not -ENOMEM,
 * so that a failure on a sparse tmpfs mapping will give SIGBUS not OOM.
 */
static inline int shmem_acct_blocks(unsigned long flags, long pages)
{
	if (!(flags & SHMEM_F_NORESERVE))
		return 0;

	return security_vm_enough_memory_mm(current->mm,
			pages * VM_ACCT(PAGE_SIZE));
}

static inline void shmem_unacct_blocks(unsigned long flags, long pages)
{
	if (flags & SHMEM_F_NORESERVE)
		vm_unacct_memory(pages * VM_ACCT(PAGE_SIZE));
}

int shmem_inode_acct_blocks(struct inode *inode, long pages)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	int err = -ENOSPC;

	if (shmem_acct_blocks(info->flags, pages))
		return err;

	might_sleep();	/* when quotas */
	if (sbinfo->max_blocks) {
		if (!percpu_counter_limited_add(&sbinfo->used_blocks,
						sbinfo->max_blocks, pages))
			goto unacct;

		err = dquot_alloc_block_nodirty(inode, pages);
		if (err) {
			percpu_counter_sub(&sbinfo->used_blocks, pages);
			goto unacct;
		}
	} else {
		err = dquot_alloc_block_nodirty(inode, pages);
		if (err)
			goto unacct;
	}

	return 0;

unacct:
	shmem_unacct_blocks(info->flags, pages);
	return err;
}

static void shmem_inode_unacct_blocks(struct inode *inode, long pages)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);

	might_sleep();	/* when quotas */
	dquot_free_block_nodirty(inode, pages);

	if (sbinfo->max_blocks)
		percpu_counter_sub(&sbinfo->used_blocks, pages);
	shmem_unacct_blocks(info->flags, pages);
}

static const struct super_operations shmem_ops;
static const struct address_space_operations shmem_aops;
static const struct file_operations shmem_file_operations;
static const struct inode_operations shmem_inode_operations;
static const struct inode_operations shmem_dir_inode_operations;
static const struct inode_operations shmem_special_inode_operations;
static const struct vm_operations_struct shmem_vm_ops;
static const struct vm_operations_struct shmem_anon_vm_ops;
static struct file_system_type shmem_fs_type;

bool shmem_mapping(const struct address_space *mapping)
{
	return mapping->a_ops == &shmem_aops;
}
EXPORT_SYMBOL_GPL(shmem_mapping);

bool vma_is_anon_shmem(const struct vm_area_struct *vma)
{
	return vma->vm_ops == &shmem_anon_vm_ops;
}

bool vma_is_shmem(const struct vm_area_struct *vma)
{
	return vma_is_anon_shmem(vma) || vma->vm_ops == &shmem_vm_ops;
}

static LIST_HEAD(shmem_swaplist);
static DEFINE_SPINLOCK(shmem_swaplist_lock);

#ifdef CONFIG_TMPFS_QUOTA

static int shmem_enable_quotas(struct super_block *sb,
			       unsigned short quota_types)
{
	int type, err = 0;

	sb_dqopt(sb)->flags |= DQUOT_QUOTA_SYS_FILE | DQUOT_NOLIST_DIRTY;
	for (type = 0; type < SHMEM_MAXQUOTAS; type++) {
		if (!(quota_types & (1 << type)))
			continue;
		err = dquot_load_quota_sb(sb, type, QFMT_SHMEM,
					  DQUOT_USAGE_ENABLED |
					  DQUOT_LIMITS_ENABLED);
		if (err)
			goto out_err;
	}
	return 0;

out_err:
	pr_warn("tmpfs: failed to enable quota tracking (type=%d, err=%d)\n",
		type, err);
	for (type--; type >= 0; type--)
		dquot_quota_off(sb, type);
	return err;
}

static void shmem_disable_quotas(struct super_block *sb)
{
	int type;

	for (type = 0; type < SHMEM_MAXQUOTAS; type++)
		dquot_quota_off(sb, type);
}

static struct dquot __rcu **shmem_get_dquots(struct inode *inode)
{
	return SHMEM_I(inode)->i_dquot;
}
#endif /* CONFIG_TMPFS_QUOTA */

/*
 * shmem_reserve_inode() performs bookkeeping to reserve a shmem inode, and
 * produces a novel ino for the newly allocated inode.
 *
 * It may also be called when making a hard link to permit the space needed by
 * each dentry. However, in that case, no new inode number is needed since that
 * internally draws from another pool of inode numbers (currently global
 * get_next_ino()). This case is indicated by passing NULL as inop.
 */
#define SHMEM_INO_BATCH 1024
static int shmem_reserve_inode(struct super_block *sb, ino_t *inop)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);
	ino_t ino;

	if (!(sb->s_flags & SB_KERNMOUNT)) {
		raw_spin_lock(&sbinfo->stat_lock);
		if (sbinfo->max_inodes) {
			if (sbinfo->free_ispace < BOGO_INODE_SIZE) {
				raw_spin_unlock(&sbinfo->stat_lock);
				return -ENOSPC;
			}
			sbinfo->free_ispace -= BOGO_INODE_SIZE;
		}
		if (inop) {
			ino = sbinfo->next_ino++;
			if (unlikely(is_zero_ino(ino)))
				ino = sbinfo->next_ino++;
			if (unlikely(!sbinfo->full_inums &&
				     ino > UINT_MAX)) {
				/*
				 * Emulate get_next_ino uint wraparound for
				 * compatibility
				 */
				if (IS_ENABLED(CONFIG_64BIT))
					pr_warn("%s: inode number overflow on device %d, consider using inode64 mount option\n",
						__func__, MINOR(sb->s_dev));
				sbinfo->next_ino = 1;
				ino = sbinfo->next_ino++;
			}
			*inop = ino;
		}
		raw_spin_unlock(&sbinfo->stat_lock);
	} else if (inop) {
		/*
		 * __shmem_file_setup, one of our callers, is lock-free: it
		 * doesn't hold stat_lock in shmem_reserve_inode since
		 * max_inodes is always 0, and is called from potentially
		 * unknown contexts. As such, use a per-cpu batched allocator
		 * which doesn't require the per-sb stat_lock unless we are at
		 * the batch boundary.
		 *
		 * We don't need to worry about inode{32,64} since SB_KERNMOUNT
		 * shmem mounts are not exposed to userspace, so we don't need
		 * to worry about things like glibc compatibility.
		 */
		ino_t *next_ino;

		next_ino = per_cpu_ptr(sbinfo->ino_batch, get_cpu());
		ino = *next_ino;
		if (unlikely(ino % SHMEM_INO_BATCH == 0)) {
			raw_spin_lock(&sbinfo->stat_lock);
			ino = sbinfo->next_ino;
			sbinfo->next_ino += SHMEM_INO_BATCH;
			raw_spin_unlock(&sbinfo->stat_lock);
			if (unlikely(is_zero_ino(ino)))
				ino++;
		}
		*inop = ino;
		*next_ino = ++ino;
		put_cpu();
	}

	return 0;
}

static void shmem_free_inode(struct super_block *sb, size_t freed_ispace)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);
	if (sbinfo->max_inodes) {
		raw_spin_lock(&sbinfo->stat_lock);
		sbinfo->free_ispace += BOGO_INODE_SIZE + freed_ispace;
		raw_spin_unlock(&sbinfo->stat_lock);
	}
}

/**
 * shmem_recalc_inode - recalculate the block usage of an inode
 * @inode: inode to recalc
 * @alloced: the change in number of pages allocated to inode
 * @swapped: the change in number of pages swapped from inode
 *
 * We have to calculate the free blocks since the mm can drop
 * undirtied hole pages behind our back.
 *
 * But normally   info->alloced == inode->i_mapping->nrpages + info->swapped
 * So mm freed is info->alloced - (inode->i_mapping->nrpages + info->swapped)
 *
 * Return: true if swapped was incremented from 0, for shmem_writeout().
 */
bool shmem_recalc_inode(struct inode *inode, long alloced, long swapped)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	bool first_swapped = false;
	long freed;

	spin_lock(&info->lock);
	info->alloced += alloced;
	info->swapped += swapped;
	freed = info->alloced - info->swapped -
		READ_ONCE(inode->i_mapping->nrpages);
	/*
	 * Special case: whereas normally shmem_recalc_inode() is called
	 * after i_mapping->nrpages has already been adjusted (up or down),
	 * shmem_writeout() has to raise swapped before nrpages is lowered -
	 * to stop a racing shmem_recalc_inode() from thinking that a page has
	 * been freed.  Compensate here, to avoid the need for a followup call.
	 */
	if (swapped > 0) {
		if (info->swapped == swapped)
			first_swapped = true;
		freed += swapped;
	}
	if (freed > 0)
		info->alloced -= freed;
	spin_unlock(&info->lock);

	/* The quota case may block */
	if (freed > 0)
		shmem_inode_unacct_blocks(inode, freed);
	return first_swapped;
}

bool shmem_charge(struct inode *inode, long pages)
{
	struct address_space *mapping = inode->i_mapping;

	if (shmem_inode_acct_blocks(inode, pages))
		return false;

	/* nrpages adjustment first, then shmem_recalc_inode() when balanced */
	xa_lock_irq(&mapping->i_pages);
	mapping->nrpages += pages;
	xa_unlock_irq(&mapping->i_pages);

	shmem_recalc_inode(inode, pages, 0);
	return true;
}

void shmem_uncharge(struct inode *inode, long pages)
{
	/* pages argument is currently unused: keep it to help debugging */
	/* nrpages adjustment done by __filemap_remove_folio() or caller */

	shmem_recalc_inode(inode, 0, 0);
}

/*
 * Replace item expected in xarray by a new item, while holding xa_lock.
 */
static int shmem_replace_entry(struct address_space *mapping,
			pgoff_t index, void *expected, void *replacement)
{
	XA_STATE(xas, &mapping->i_pages, index);
	void *item;

	VM_BUG_ON(!expected);
	VM_BUG_ON(!replacement);
	item = xas_load(&xas);
	if (item != expected)
		return -ENOENT;
	xas_store(&xas, replacement);
	return 0;
}

/*
 * Sometimes, before we decide whether to proceed or to fail, we must check
 * that an entry was not already brought back or split by a racing thread.
 *
 * Checking folio is not enough: by the time a swapcache folio is locked, it
 * might be reused, and again be swapcache, using the same swap as before.
 * Returns the swap entry's order if it still presents, else returns -1.
 */
static int shmem_confirm_swap(struct address_space *mapping, pgoff_t index,
			      swp_entry_t swap)
{
	XA_STATE(xas, &mapping->i_pages, index);
	int ret = -1;
	void *entry;

	rcu_read_lock();
	do {
		entry = xas_load(&xas);
		if (entry == swp_to_radix_entry(swap))
			ret = xas_get_order(&xas);
	} while (xas_retry(&xas, entry));
	rcu_read_unlock();
	return ret;
}

/*
 * Definitions for "huge tmpfs": tmpfs mounted with the huge= option
 *
 * SHMEM_HUGE_NEVER:
 *	disables huge pages for the mount;
 * SHMEM_HUGE_ALWAYS:
 *	enables huge pages for the mount;
 * SHMEM_HUGE_WITHIN_SIZE:
 *	only allocate huge pages if the page will be fully within i_size,
 *	also respect madvise() hints;
 * SHMEM_HUGE_ADVISE:
 *	only allocate huge pages if requested with madvise();
 */

#define SHMEM_HUGE_NEVER	0
#define SHMEM_HUGE_ALWAYS	1
#define SHMEM_HUGE_WITHIN_SIZE	2
#define SHMEM_HUGE_ADVISE	3

/*
 * Special values.
 * Only can be set via /sys/kernel/mm/transparent_hugepage/shmem_enabled:
 *
 * SHMEM_HUGE_DENY:
 *	disables huge on shm_mnt and all mounts, for emergency use;
 * SHMEM_HUGE_FORCE:
 *	enables huge on shm_mnt and all mounts, w/o needing option, for testing;
 *
 */
#define SHMEM_HUGE_DENY		(-1)
#define SHMEM_HUGE_FORCE	(-2)

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/* ifdef here to avoid bloating shmem.o when not necessary */

#if defined(CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_NEVER)
#define SHMEM_HUGE_DEFAULT SHMEM_HUGE_NEVER
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_ALWAYS)
#define SHMEM_HUGE_DEFAULT SHMEM_HUGE_ALWAYS
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_WITHIN_SIZE)
#define SHMEM_HUGE_DEFAULT SHMEM_HUGE_WITHIN_SIZE
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_SHMEM_HUGE_ADVISE)
#define SHMEM_HUGE_DEFAULT SHMEM_HUGE_ADVISE
#else
#define SHMEM_HUGE_DEFAULT SHMEM_HUGE_NEVER
#endif

static int shmem_huge __read_mostly = SHMEM_HUGE_DEFAULT;

#undef SHMEM_HUGE_DEFAULT

#if defined(CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_NEVER)
#define TMPFS_HUGE_DEFAULT SHMEM_HUGE_NEVER
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_ALWAYS)
#define TMPFS_HUGE_DEFAULT SHMEM_HUGE_ALWAYS
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_WITHIN_SIZE)
#define TMPFS_HUGE_DEFAULT SHMEM_HUGE_WITHIN_SIZE
#elif defined(CONFIG_TRANSPARENT_HUGEPAGE_TMPFS_HUGE_ADVISE)
#define TMPFS_HUGE_DEFAULT SHMEM_HUGE_ADVISE
#else
#define TMPFS_HUGE_DEFAULT SHMEM_HUGE_NEVER
#endif

static int tmpfs_huge __read_mostly = TMPFS_HUGE_DEFAULT;

#undef TMPFS_HUGE_DEFAULT

static unsigned int shmem_get_orders_within_size(struct inode *inode,
		unsigned long within_size_orders, pgoff_t index,
		loff_t write_end)
{
	pgoff_t aligned_index;
	unsigned long order;
	loff_t i_size;

	order = highest_order(within_size_orders);
	while (within_size_orders) {
		aligned_index = round_up(index + 1, 1 << order);
		i_size = max(write_end, i_size_read(inode));
		i_size = round_up(i_size, PAGE_SIZE);
		if (i_size >> PAGE_SHIFT >= aligned_index)
			return within_size_orders;

		order = next_order(&within_size_orders, order);
	}

	return 0;
}

static unsigned int shmem_huge_global_enabled(struct inode *inode, pgoff_t index,
					      loff_t write_end, bool shmem_huge_force,
					      struct vm_area_struct *vma,
					      vm_flags_t vm_flags)
{
	unsigned int maybe_pmd_order = HPAGE_PMD_ORDER > MAX_PAGECACHE_ORDER ?
		0 : BIT(HPAGE_PMD_ORDER);
	unsigned long within_size_orders;

	if (!S_ISREG(inode->i_mode))
		return 0;
	if (shmem_huge == SHMEM_HUGE_DENY)
		return 0;
	if (shmem_huge_force || shmem_huge == SHMEM_HUGE_FORCE)
		return maybe_pmd_order;

	/*
	 * The huge order allocation for anon shmem is controlled through
	 * the mTHP interface, so we still use PMD-sized huge order to
	 * check whether global control is enabled.
	 *
	 * For tmpfs with 'huge=always' or 'huge=within_size' mount option,
	 * we will always try PMD-sized order first. If that failed, it will
	 * fall back to small large folios.
	 */
	switch (SHMEM_SB(inode->i_sb)->huge) {
	case SHMEM_HUGE_ALWAYS:
		return THP_ORDERS_ALL_FILE_DEFAULT;
	case SHMEM_HUGE_WITHIN_SIZE:
		within_size_orders = shmem_get_orders_within_size(inode,
				THP_ORDERS_ALL_FILE_DEFAULT, index, write_end);
		if (within_size_orders > 0)
			return within_size_orders;

		fallthrough;
	case SHMEM_HUGE_ADVISE:
		if (vm_flags & VM_HUGEPAGE)
			return THP_ORDERS_ALL_FILE_DEFAULT;
		fallthrough;
	default:
		return 0;
	}
}

static int shmem_parse_huge(const char *str)
{
	int huge;

	if (!str)
		return -EINVAL;

	if (!strcmp(str, "never"))
		huge = SHMEM_HUGE_NEVER;
	else if (!strcmp(str, "always"))
		huge = SHMEM_HUGE_ALWAYS;
	else if (!strcmp(str, "within_size"))
		huge = SHMEM_HUGE_WITHIN_SIZE;
	else if (!strcmp(str, "advise"))
		huge = SHMEM_HUGE_ADVISE;
	else if (!strcmp(str, "deny"))
		huge = SHMEM_HUGE_DENY;
	else if (!strcmp(str, "force"))
		huge = SHMEM_HUGE_FORCE;
	else
		return -EINVAL;

	if (!has_transparent_hugepage() &&
	    huge != SHMEM_HUGE_NEVER && huge != SHMEM_HUGE_DENY)
		return -EINVAL;

	/* Do not override huge allocation policy with non-PMD sized mTHP */
	if (huge == SHMEM_HUGE_FORCE &&
	    huge_shmem_orders_inherit != BIT(HPAGE_PMD_ORDER))
		return -EINVAL;

	return huge;
}

#if defined(CONFIG_SYSFS) || defined(CONFIG_TMPFS)
static const char *shmem_format_huge(int huge)
{
	switch (huge) {
	case SHMEM_HUGE_NEVER:
		return "never";
	case SHMEM_HUGE_ALWAYS:
		return "always";
	case SHMEM_HUGE_WITHIN_SIZE:
		return "within_size";
	case SHMEM_HUGE_ADVISE:
		return "advise";
	case SHMEM_HUGE_DENY:
		return "deny";
	case SHMEM_HUGE_FORCE:
		return "force";
	default:
		VM_BUG_ON(1);
		return "bad_val";
	}
}
#endif

static unsigned long shmem_unused_huge_shrink(struct shmem_sb_info *sbinfo,
		struct shrink_control *sc, unsigned long nr_to_free)
{
	LIST_HEAD(list), *pos, *next;
	struct inode *inode;
	struct shmem_inode_info *info;
	struct folio *folio;
	unsigned long batch = sc ? sc->nr_to_scan : 128;
	unsigned long split = 0, freed = 0;

	if (list_empty(&sbinfo->shrinklist))
		return SHRINK_STOP;

	spin_lock(&sbinfo->shrinklist_lock);
	list_for_each_safe(pos, next, &sbinfo->shrinklist) {
		info = list_entry(pos, struct shmem_inode_info, shrinklist);

		/* pin the inode */
		inode = igrab(&info->vfs_inode);

		/* inode is about to be evicted */
		if (!inode) {
			list_del_init(&info->shrinklist);
			goto next;
		}

		list_move(&info->shrinklist, &list);
next:
		sbinfo->shrinklist_len--;
		if (!--batch)
			break;
	}
	spin_unlock(&sbinfo->shrinklist_lock);

	list_for_each_safe(pos, next, &list) {
		pgoff_t next, end;
		loff_t i_size;
		int ret;

		info = list_entry(pos, struct shmem_inode_info, shrinklist);
		inode = &info->vfs_inode;

		if (nr_to_free && freed >= nr_to_free)
			goto move_back;

		i_size = i_size_read(inode);
		folio = filemap_get_entry(inode->i_mapping, i_size / PAGE_SIZE);
		if (!folio || xa_is_value(folio))
			goto drop;

		/* No large folio at the end of the file: nothing to split */
		if (!folio_test_large(folio)) {
			folio_put(folio);
			goto drop;
		}

		/* Check if there is anything to gain from splitting */
		next = folio_next_index(folio);
		end = shmem_fallocend(inode, DIV_ROUND_UP(i_size, PAGE_SIZE));
		if (end <= folio->index || end >= next) {
			folio_put(folio);
			goto drop;
		}

		/*
		 * Move the inode on the list back to shrinklist if we failed
		 * to lock the page at this time.
		 *
		 * Waiting for the lock may lead to deadlock in the
		 * reclaim path.
		 */
		if (!folio_trylock(folio)) {
			folio_put(folio);
			goto move_back;
		}

		ret = split_folio(folio);
		folio_unlock(folio);
		folio_put(folio);

		/* If split failed move the inode on the list back to shrinklist */
		if (ret)
			goto move_back;

		freed += next - end;
		split++;
drop:
		list_del_init(&info->shrinklist);
		goto put;
move_back:
		/*
		 * Make sure the inode is either on the global list or deleted
		 * from any local list before iput() since it could be deleted
		 * in another thread once we put the inode (then the local list
		 * is corrupted).
		 */
		spin_lock(&sbinfo->shrinklist_lock);
		list_move(&info->shrinklist, &sbinfo->shrinklist);
		sbinfo->shrinklist_len++;
		spin_unlock(&sbinfo->shrinklist_lock);
put:
		iput(inode);
	}

	return split;
}

static long shmem_unused_huge_scan(struct super_block *sb,
		struct shrink_control *sc)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);

	if (!READ_ONCE(sbinfo->shrinklist_len))
		return SHRINK_STOP;

	return shmem_unused_huge_shrink(sbinfo, sc, 0);
}

static long shmem_unused_huge_count(struct super_block *sb,
		struct shrink_control *sc)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);
	return READ_ONCE(sbinfo->shrinklist_len);
}
#else /* !CONFIG_TRANSPARENT_HUGEPAGE */

#define shmem_huge SHMEM_HUGE_DENY

static unsigned long shmem_unused_huge_shrink(struct shmem_sb_info *sbinfo,
		struct shrink_control *sc, unsigned long nr_to_free)
{
	return 0;
}

static unsigned int shmem_huge_global_enabled(struct inode *inode, pgoff_t index,
					      loff_t write_end, bool shmem_huge_force,
					      struct vm_area_struct *vma,
					      vm_flags_t vm_flags)
{
	return 0;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */

static void shmem_update_stats(struct folio *folio, int nr_pages)
{
	if (folio_test_pmd_mappable(folio))
		lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, nr_pages);
	lruvec_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
	lruvec_stat_mod_folio(folio, NR_SHMEM, nr_pages);
}

/*
 * Somewhat like filemap_add_folio, but error if expected item has gone.
 */
int shmem_add_to_page_cache(struct folio *folio,
			    struct address_space *mapping,
			    pgoff_t index, void *expected, gfp_t gfp)
{
	XA_STATE_ORDER(xas, &mapping->i_pages, index, folio_order(folio));
	unsigned long nr = folio_nr_pages(folio);
	swp_entry_t iter, swap;
	void *entry;

	VM_BUG_ON_FOLIO(index != round_down(index, nr), folio);
	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
	VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio);

	folio_ref_add(folio, nr);
	folio->mapping = mapping;
	folio->index = index;

	gfp &= GFP_RECLAIM_MASK;
	folio_throttle_swaprate(folio, gfp);
	swap = radix_to_swp_entry(expected);

	do {
		iter = swap;
		xas_lock_irq(&xas);
		xas_for_each_conflict(&xas, entry) {
			/*
			 * The range must either be empty, or filled with
			 * expected swap entries. Shmem swap entries are never
			 * partially freed without split of both entry and
			 * folio, so there shouldn't be any holes.
			 */
			if (!expected || entry != swp_to_radix_entry(iter)) {
				xas_set_err(&xas, -EEXIST);
				goto unlock;
			}
			iter.val += 1 << xas_get_order(&xas);
		}
		if (expected && iter.val - nr != swap.val) {
			xas_set_err(&xas, -EEXIST);
			goto unlock;
		}
		xas_store(&xas, folio);
		if (xas_error(&xas))
			goto unlock;
		shmem_update_stats(folio, nr);
		mapping->nrpages += nr;
unlock:
		xas_unlock_irq(&xas);
	} while (xas_nomem(&xas, gfp));

	if (xas_error(&xas)) {
		folio->mapping = NULL;
		folio_ref_sub(folio, nr);
		return xas_error(&xas);
	}

	return 0;
}

/*
 * Somewhat like filemap_remove_folio, but substitutes swap for @folio.
 */
static void shmem_delete_from_page_cache(struct folio *folio, void *radswap)
{
	struct address_space *mapping = folio->mapping;
	long nr = folio_nr_pages(folio);
	int error;

	xa_lock_irq(&mapping->i_pages);
	error = shmem_replace_entry(mapping, folio->index, folio, radswap);
	folio->mapping = NULL;
	mapping->nrpages -= nr;
	shmem_update_stats(folio, -nr);
	xa_unlock_irq(&mapping->i_pages);
	folio_put_refs(folio, nr);
	BUG_ON(error);
}

/*
 * Remove swap entry from page cache, free the swap and its page cache. Returns
 * the number of pages being freed. 0 means entry not found in XArray (0 pages
 * being freed).
 */
static long shmem_free_swap(struct address_space *mapping,
			    pgoff_t index, pgoff_t end, void *radswap)
{
	XA_STATE(xas, &mapping->i_pages, index);
	unsigned int nr_pages = 0;
	pgoff_t base;
	void *entry;

	xas_lock_irq(&xas);
	entry = xas_load(&xas);
	if (entry == radswap) {
		nr_pages = 1 << xas_get_order(&xas);
		base = round_down(xas.xa_index, nr_pages);
		if (base < index || base + nr_pages - 1 > end)
			nr_pages = 0;
		else
			xas_store(&xas, NULL);
	}
	xas_unlock_irq(&xas);

	if (nr_pages)
		swap_put_entries_direct(radix_to_swp_entry(radswap), nr_pages);

	return nr_pages;
}

/*
 * Determine (in bytes) how many of the shmem object's pages mapped by the
 * given offsets are swapped out.
 *
 * This is safe to call without i_rwsem or the i_pages lock thanks to RCU,
 * as long as the inode doesn't go away and racy results are not a problem.
 */
unsigned long shmem_partial_swap_usage(struct address_space *mapping,
						pgoff_t start, pgoff_t end)
{
	XA_STATE(xas, &mapping->i_pages, start);
	struct folio *folio;
	unsigned long swapped = 0;
	unsigned long max = end - 1;

	rcu_read_lock();
	xas_for_each(&xas, folio, max) {
		if (xas_retry(&xas, folio))
			continue;
		if (xa_is_value(folio))
			swapped += 1 << xas_get_order(&xas);
		if (xas.xa_index == max)
			break;
		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}
	}
	rcu_read_unlock();

	return swapped << PAGE_SHIFT;
}

/*
 * Determine (in bytes) how many of the shmem object's pages mapped by the
 * given vma is swapped out.
 *
 * This is safe to call without i_rwsem or the i_pages lock thanks to RCU,
 * as long as the inode doesn't go away and racy results are not a problem.
 */
unsigned long shmem_swap_usage(struct vm_area_struct *vma)
{
	struct inode *inode = file_inode(vma->vm_file);
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct address_space *mapping = inode->i_mapping;
	unsigned long swapped;

	/* Be careful as we don't hold info->lock */
	swapped = READ_ONCE(info->swapped);

	/*
	 * The easier cases are when the shmem object has nothing in swap, or
	 * the vma maps it whole. Then we can simply use the stats that we
	 * already track.
	 */
	if (!swapped)
		return 0;

	if (!vma->vm_pgoff && vma->vm_end - vma->vm_start >= inode->i_size)
		return swapped << PAGE_SHIFT;

	/* Here comes the more involved part */
	return shmem_partial_swap_usage(mapping, vma->vm_pgoff,
					vma->vm_pgoff + vma_pages(vma));
}

/*
 * SysV IPC SHM_UNLOCK restore Unevictable pages to their evictable lists.
 */
void shmem_unlock_mapping(struct address_space *mapping)
{
	struct folio_batch fbatch;
	pgoff_t index = 0;

	folio_batch_init(&fbatch);
	/*
	 * Minor point, but we might as well stop if someone else SHM_LOCKs it.
	 */
	while (!mapping_unevictable(mapping) &&
	       filemap_get_folios(mapping, &index, ~0UL, &fbatch)) {
		check_move_unevictable_folios(&fbatch);
		folio_batch_release(&fbatch);
		cond_resched();
	}
}

static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index)
{
	struct folio *folio;

	/*
	 * At first avoid shmem_get_folio(,,,SGP_READ): that fails
	 * beyond i_size, and reports fallocated folios as holes.
	 */
	folio = filemap_get_entry(inode->i_mapping, index);
	if (!folio)
		return folio;
	if (!xa_is_value(folio)) {
		folio_lock(folio);
		if (folio->mapping == inode->i_mapping)
			return folio;
		/* The folio has been swapped out */
		folio_unlock(folio);
		folio_put(folio);
	}
	/*
	 * But read a folio back from swap if any of it is within i_size
	 * (although in some cases this is just a waste of time).
	 */
	folio = NULL;
	shmem_get_folio(inode, index, 0, &folio, SGP_READ);
	return folio;
}

/*
 * Remove range of pages and swap entries from page cache, and free them.
 * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate.
 */
static void shmem_undo_range(struct inode *inode, loff_t lstart, uoff_t lend,
								 bool unfalloc)
{
	struct address_space *mapping = inode->i_mapping;
	struct shmem_inode_info *info = SHMEM_I(inode);
	pgoff_t start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
	pgoff_t end = (lend + 1) >> PAGE_SHIFT;
	struct folio_batch fbatch;
	pgoff_t indices[PAGEVEC_SIZE];
	struct folio *folio;
	bool same_folio;
	long nr_swaps_freed = 0;
	pgoff_t index;
	int i;

	if (lend == -1)
		end = -1;	/* unsigned, so actually very big */

	if (info->fallocend > start && info->fallocend <= end && !unfalloc)
		info->fallocend = start;

	folio_batch_init(&fbatch);
	index = start;
	while (index < end && find_lock_entries(mapping, &index, end - 1,
			&fbatch, indices)) {
		for (i = 0; i < folio_batch_count(&fbatch); i++) {
			folio = fbatch.folios[i];

			if (xa_is_value(folio)) {
				if (unfalloc)
					continue;
				nr_swaps_freed += shmem_free_swap(mapping, indices[i],
								  end - 1, folio);
				continue;
			}

			if (!unfalloc || !folio_test_uptodate(folio))
				truncate_inode_folio(mapping, folio);
			folio_unlock(folio);
		}
		folio_batch_remove_exceptionals(&fbatch);
		folio_batch_release(&fbatch);
		cond_resched();
	}

	/*
	 * When undoing a failed fallocate, we want none of the partial folio
	 * zeroing and splitting below, but shall want to truncate the whole
	 * folio when !uptodate indicates that it was added by this fallocate,
	 * even when [lstart, lend] covers only a part of the folio.
	 */
	if (unfalloc)
		goto whole_folios;

	same_folio = (lstart >> PAGE_SHIFT) == (lend >> PAGE_SHIFT);
	folio = shmem_get_partial_folio(inode, lstart >> PAGE_SHIFT);
	if (folio) {
		same_folio = lend < folio_next_pos(folio);
		folio_mark_dirty(folio);
		if (!truncate_inode_partial_folio(folio, lstart, lend)) {
			start = folio_next_index(folio);
			if (same_folio)
				end = folio->index;
		}
		folio_unlock(folio);
		folio_put(folio);
		folio = NULL;
	}

	if (!same_folio)
		folio = shmem_get_partial_folio(inode, lend >> PAGE_SHIFT);
	if (folio) {
		folio_mark_dirty(folio);
		if (!truncate_inode_partial_folio(folio, lstart, lend))
			end = folio->index;
		folio_unlock(folio);
		folio_put(folio);
	}

whole_folios:

	index = start;
	while (index < end) {
		cond_resched();

		if (!find_get_entries(mapping, &index, end - 1, &fbatch,
				indices)) {
			/* If all gone or hole-punch or unfalloc, we're done */
			if (index == start || end != -1)
				break;
			/* But if truncating, restart to make sure all gone */
			index = start;
			continue;
		}
		for (i = 0; i < folio_batch_count(&fbatch); i++) {
			folio = fbatch.folios[i];

			if (xa_is_value(folio)) {
				int order;
				long swaps_freed;

				if (unfalloc)
					continue;
				swaps_freed = shmem_free_swap(mapping, indices[i],
							      end - 1, folio);
				if (!swaps_freed) {
					pgoff_t base = indices[i];

					order = shmem_confirm_swap(mapping, indices[i],
								   radix_to_swp_entry(folio));
					/*
					 * If found a large swap entry cross the end or start
					 * border, skip it as the truncate_inode_partial_folio
					 * above should have at least zerod its content once.
					 */
					if (order > 0) {
						base = round_down(base, 1 << order);
						if (base < start || base + (1 << order) > end)
							continue;
					}
					/* Swap was replaced by page or extended, retry */
					index = base;
					break;
				}
				nr_swaps_freed += swaps_freed;
				continue;
			}

			folio_lock(folio);

			if (!unfalloc || !folio_test_uptodate(folio)) {
				if (folio_mapping(folio) != mapping) {
					/* Page was replaced by swap: retry */
					folio_unlock(folio);
					index = indices[i];
					break;
				}
				VM_BUG_ON_FOLIO(folio_test_writeback(folio),
						folio);

				if (!folio_test_large(folio)) {
					truncate_inode_folio(mapping, folio);
				} else if (truncate_inode_partial_folio(folio, lstart, lend)) {
					/*
					 * If we split a page, reset the loop so
					 * that we pick up the new sub pages.
					 * Otherwise the THP was entirely
					 * dropped or the target range was
					 * zeroed, so just continue the loop as
					 * is.
					 */
					if (!folio_test_large(folio)) {
						folio_unlock(folio);
						index = start;
						break;
					}
				}
			}
			folio_unlock(folio);
		}
		folio_batch_remove_exceptionals(&fbatch);
		folio_batch_release(&fbatch);
	}

	shmem_recalc_inode(inode, 0, -nr_swaps_freed);
}

void shmem_truncate_range(struct inode *inode, loff_t lstart, uoff_t lend)
{
	shmem_undo_range(inode, lstart, lend, false);
	inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode));
	inode_inc_iversion(inode);
}
EXPORT_SYMBOL_GPL(shmem_truncate_range);

static int shmem_getattr(struct mnt_idmap *idmap,
			 const struct path *path, struct kstat *stat,
			 u32 request_mask, unsigned int query_flags)
{
	struct inode *inode = path->dentry->d_inode;
	struct shmem_inode_info *info = SHMEM_I(inode);

	if (info->alloced - info->swapped != inode->i_mapping->nrpages)
		shmem_recalc_inode(inode, 0, 0);

	if (info->fsflags & FS_APPEND_FL)
		stat->attributes |= STATX_ATTR_APPEND;
	if (info->fsflags & FS_IMMUTABLE_FL)
		stat->attributes |= STATX_ATTR_IMMUTABLE;
	if (info->fsflags & FS_NODUMP_FL)
		stat->attributes |= STATX_ATTR_NODUMP;
	stat->attributes_mask |= (STATX_ATTR_APPEND |
			STATX_ATTR_IMMUTABLE |
			STATX_ATTR_NODUMP);
	generic_fillattr(idmap, request_mask, inode, stat);

	if (shmem_huge_global_enabled(inode, 0, 0, false, NULL, 0))
		stat->blksize = HPAGE_PMD_SIZE;

	if (request_mask & STATX_BTIME) {
		stat->result_mask |= STATX_BTIME;
		stat->btime.tv_sec = info->i_crtime.tv_sec;
		stat->btime.tv_nsec = info->i_crtime.tv_nsec;
	}

	return 0;
}

static int shmem_setattr(struct mnt_idmap *idmap,
			 struct dentry *dentry, struct iattr *attr)
{
	struct inode *inode = d_inode(dentry);
	struct shmem_inode_info *info = SHMEM_I(inode);
	int error;
	bool update_mtime = false;
	bool update_ctime = true;

	error = setattr_prepare(idmap, dentry, attr);
	if (error)
		return error;

	if ((info->seals & F_SEAL_EXEC) && (attr->ia_valid & ATTR_MODE)) {
		if ((inode->i_mode ^ attr->ia_mode) & 0111) {
			return -EPERM;
		}
	}

	if (S_ISREG(inode->i_mode) && (attr->ia_valid & ATTR_SIZE)) {
		loff_t oldsize = inode->i_size;
		loff_t newsize = attr->ia_size;

		/* protected by i_rwsem */
		if ((newsize < oldsize && (info->seals & F_SEAL_SHRINK)) ||
		    (newsize > oldsize && (info->seals & F_SEAL_GROW)))
			return -EPERM;

		if (newsize != oldsize) {
			if (info->flags & SHMEM_F_MAPPING_FROZEN)
				return -EPERM;
			error = shmem_reacct_size(SHMEM_I(inode)->flags,
					oldsize, newsize);
			if (error)
				return error;
			i_size_write(inode, newsize);
			update_mtime = true;
		} else {
			update_ctime = false;
		}
		if (newsize <= oldsize) {
			loff_t holebegin = round_up(newsize, PAGE_SIZE);
			if (oldsize > holebegin)
				unmap_mapping_range(inode->i_mapping,
							holebegin, 0, 1);
			if (info->alloced)
				shmem_truncate_range(inode,
							newsize, (loff_t)-1);
			/* unmap again to remove racily COWed private pages */
			if (oldsize > holebegin)
				unmap_mapping_range(inode->i_mapping,
							holebegin, 0, 1);
		}
	}

	if (is_quota_modification(idmap, inode, attr)) {
		error = dquot_initialize(inode);
		if (error)
			return error;
	}

	/* Transfer quota accounting */
	if (i_uid_needs_update(idmap, attr, inode) ||
	    i_gid_needs_update(idmap, attr, inode)) {
		error = dquot_transfer(idmap, inode, attr);
		if (error)
			return error;
	}

	setattr_copy(idmap, inode, attr);
	if (attr->ia_valid & ATTR_MODE)
		error = posix_acl_chmod(idmap, dentry, inode->i_mode);
	if (!error && update_ctime) {
		inode_set_ctime_current(inode);
		if (update_mtime)
			inode_set_mtime_to_ts(inode, inode_get_ctime(inode));
		inode_inc_iversion(inode);
	}
	return error;
}

static void shmem_evict_inode(struct inode *inode)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	size_t freed = 0;

	if (shmem_mapping(inode->i_mapping)) {
		shmem_unacct_size(info->flags, inode->i_size);
		inode->i_size = 0;
		mapping_set_exiting(inode->i_mapping);
		shmem_truncate_range(inode, 0, (loff_t)-1);
		if (!list_empty(&info->shrinklist)) {
			spin_lock(&sbinfo->shrinklist_lock);
			if (!list_empty(&info->shrinklist)) {
				list_del_init(&info->shrinklist);
				sbinfo->shrinklist_len--;
			}
			spin_unlock(&sbinfo->shrinklist_lock);
		}
		while (!list_empty(&info->swaplist)) {
			/* Wait while shmem_unuse() is scanning this inode... */
			wait_var_event(&info->stop_eviction,
				       !atomic_read(&info->stop_eviction));
			spin_lock(&shmem_swaplist_lock);
			/* ...but beware of the race if we peeked too early */
			if (!atomic_read(&info->stop_eviction))
				list_del_init(&info->swaplist);
			spin_unlock(&shmem_swaplist_lock);
		}
	}

	simple_xattrs_free(&info->xattrs, sbinfo->max_inodes ? &freed : NULL);
	shmem_free_inode(inode->i_sb, freed);
	WARN_ON(inode->i_blocks);
	clear_inode(inode);
#ifdef CONFIG_TMPFS_QUOTA
	dquot_free_inode(inode);
	dquot_drop(inode);
#endif
}

static unsigned int shmem_find_swap_entries(struct address_space *mapping,
				pgoff_t start, struct folio_batch *fbatch,
				pgoff_t *indices, unsigned int type)
{
	XA_STATE(xas, &mapping->i_pages, start);
	struct folio *folio;
	swp_entry_t entry;

	rcu_read_lock();
	xas_for_each(&xas, folio, ULONG_MAX) {
		if (xas_retry(&xas, folio))
			continue;

		if (!xa_is_value(folio))
			continue;

		entry = radix_to_swp_entry(folio);
		/*
		 * swapin error entries can be found in the mapping. But they're
		 * deliberately ignored here as we've done everything we can do.
		 */
		if (swp_type(entry) != type)
			continue;

		indices[folio_batch_count(fbatch)] = xas.xa_index;
		if (!folio_batch_add(fbatch, folio))
			break;

		if (need_resched()) {
			xas_pause(&xas);
			cond_resched_rcu();
		}
	}
	rcu_read_unlock();

	return folio_batch_count(fbatch);
}

/*
 * Move the swapped pages for an inode to page cache. Returns the count
 * of pages swapped in, or the error in case of failure.
 */
static int shmem_unuse_swap_entries(struct inode *inode,
		struct folio_batch *fbatch, pgoff_t *indices)
{
	int i = 0;
	int ret = 0;
	int error = 0;
	struct address_space *mapping = inode->i_mapping;

	for (i = 0; i < folio_batch_count(fbatch); i++) {
		struct folio *folio = fbatch->folios[i];

		error = shmem_swapin_folio(inode, indices[i], &folio, SGP_CACHE,
					mapping_gfp_mask(mapping), NULL, NULL);
		if (error == 0) {
			folio_unlock(folio);
			folio_put(folio);
			ret++;
		}
		if (error == -ENOMEM)
			break;
		error = 0;
	}
	return error ? error : ret;
}

/*
 * If swap found in inode, free it and move page from swapcache to filecache.
 */
static int shmem_unuse_inode(struct inode *inode, unsigned int type)
{
	struct address_space *mapping = inode->i_mapping;
	pgoff_t start = 0;
	struct folio_batch fbatch;
	pgoff_t indices[PAGEVEC_SIZE];
	int ret = 0;

	do {
		folio_batch_init(&fbatch);
		if (!shmem_find_swap_entries(mapping, start, &fbatch,
					     indices, type)) {
			ret = 0;
			break;
		}

		ret = shmem_unuse_swap_entries(inode, &fbatch, indices);
		if (ret < 0)
			break;

		start = indices[folio_batch_count(&fbatch) - 1];
	} while (true);

	return ret;
}

/*
 * Read all the shared memory data that resides in the swap
 * device 'type' back into memory, so the swap device can be
 * unused.
 */
int shmem_unuse(unsigned int type)
{
	struct shmem_inode_info *info, *next;
	int error = 0;

	if (list_empty(&shmem_swaplist))
		return 0;

	spin_lock(&shmem_swaplist_lock);
start_over:
	list_for_each_entry_safe(info, next, &shmem_swaplist, swaplist) {
		if (!info->swapped) {
			list_del_init(&info->swaplist);
			continue;
		}
		/*
		 * Drop the swaplist mutex while searching the inode for swap;
		 * but before doing so, make sure shmem_evict_inode() will not
		 * remove placeholder inode from swaplist, nor let it be freed
		 * (igrab() would protect from unlink, but not from unmount).
		 */
		atomic_inc(&info->stop_eviction);
		spin_unlock(&shmem_swaplist_lock);

		error = shmem_unuse_inode(&info->vfs_inode, type);
		cond_resched();

		spin_lock(&shmem_swaplist_lock);
		if (atomic_dec_and_test(&info->stop_eviction))
			wake_up_var(&info->stop_eviction);
		if (error)
			break;
		if (list_empty(&info->swaplist))
			goto start_over;
		next = list_next_entry(info, swaplist);
		if (!info->swapped)
			list_del_init(&info->swaplist);
	}
	spin_unlock(&shmem_swaplist_lock);

	return error;
}

/**
 * shmem_writeout - Write the folio to swap
 * @folio: The folio to write
 * @plug: swap plug
 * @folio_list: list to put back folios on split
 *
 * Move the folio from the page cache to the swap cache.
 */
int shmem_writeout(struct folio *folio, struct swap_iocb **plug,
		struct list_head *folio_list)
{
	struct address_space *mapping = folio->mapping;
	struct inode *inode = mapping->host;
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	pgoff_t index;
	int nr_pages;
	bool split = false;

	if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap)
		goto redirty;

	if (!total_swap_pages)
		goto redirty;

	/*
	 * If CONFIG_THP_SWAP is not enabled, the large folio should be
	 * split when swapping.
	 *
	 * And shrinkage of pages beyond i_size does not split swap, so
	 * swapout of a large folio crossing i_size needs to split too
	 * (unless fallocate has been used to preallocate beyond EOF).
	 */
	if (folio_test_large(folio)) {
		index = shmem_fallocend(inode,
			DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE));
		if ((index > folio->index && index < folio_next_index(folio)) ||
		    !IS_ENABLED(CONFIG_THP_SWAP))
			split = true;
	}

	if (split) {
		int order;

try_split:
		order = folio_order(folio);
		/* Ensure the subpages are still dirty */
		folio_test_set_dirty(folio);
		if (split_folio_to_list(folio, folio_list))
			goto redirty;

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
		if (order >= HPAGE_PMD_ORDER) {
			count_memcg_folio_events(folio, THP_SWPOUT_FALLBACK, 1);
			count_vm_event(THP_SWPOUT_FALLBACK);
		}
#endif
		count_mthp_stat(order, MTHP_STAT_SWPOUT_FALLBACK);

		folio_clear_dirty(folio);
	}

	index = folio->index;
	nr_pages = folio_nr_pages(folio);

	/*
	 * This is somewhat ridiculous, but without plumbing a SWAP_MAP_FALLOC
	 * value into swapfile.c, the only way we can correctly account for a
	 * fallocated folio arriving here is now to initialize it and write it.
	 *
	 * That's okay for a folio already fallocated earlier, but if we have
	 * not yet completed the fallocation, then (a) we want to keep track
	 * of this folio in case we have to undo it, and (b) it may not be a
	 * good idea to continue anyway, once we're pushing into swap.  So
	 * reactivate the folio, and let shmem_fallocate() quit when too many.
	 */
	if (!folio_test_uptodate(folio)) {
		if (inode->i_private) {
			struct shmem_falloc *shmem_falloc;
			spin_lock(&inode->i_lock);
			shmem_falloc = inode->i_private;
			if (shmem_falloc &&
			    !shmem_falloc->waitq &&
			    index >= shmem_falloc->start &&
			    index < shmem_falloc->next)
				shmem_falloc->nr_unswapped += nr_pages;
			else
				shmem_falloc = NULL;
			spin_unlock(&inode->i_lock);
			if (shmem_falloc)
				goto redirty;
		}
		folio_zero_range(folio, 0, folio_size(folio));
		flush_dcache_folio(folio);
		folio_mark_uptodate(folio);
	}

	if (!folio_alloc_swap(folio)) {
		bool first_swapped = shmem_recalc_inode(inode, 0, nr_pages);
		int error;

		/*
		 * Add inode to shmem_unuse()'s list of swapped-out inodes,
		 * if it's not already there.  Do it now before the folio is
		 * removed from page cache, when its pagelock no longer
		 * protects the inode from eviction.  And do it now, after
		 * we've incremented swapped, because shmem_unuse() will
		 * prune a !swapped inode from the swaplist.
		 */
		if (first_swapped) {
			spin_lock(&shmem_swaplist_lock);
			if (list_empty(&info->swaplist))
				list_add(&info->swaplist, &shmem_swaplist);
			spin_unlock(&shmem_swaplist_lock);
		}

		folio_dup_swap(folio, NULL);
		shmem_delete_from_page_cache(folio, swp_to_radix_entry(folio->swap));

		BUG_ON(folio_mapped(folio));
		error = swap_writeout(folio, plug);
		if (error != AOP_WRITEPAGE_ACTIVATE) {
			/* folio has been unlocked */
			return error;
		}

		/*
		 * The intention here is to avoid holding on to the swap when
		 * zswap was unable to compress and unable to writeback; but
		 * it will be appropriate if other reactivate cases are added.
		 */
		error = shmem_add_to_page_cache(folio, mapping, index,
				swp_to_radix_entry(folio->swap),
				__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
		/* Swap entry might be erased by racing shmem_free_swap() */
		if (!error) {
			shmem_recalc_inode(inode, 0, -nr_pages);
			folio_put_swap(folio, NULL);
		}

		/*
		 * The swap_cache_del_folio() below could be left for
		 * shrink_folio_list()'s folio_free_swap() to dispose of;
		 * but I'm a little nervous about letting this folio out of
		 * shmem_writeout() in a hybrid half-tmpfs-half-swap state
		 * e.g. folio_mapping(folio) might give an unexpected answer.
		 */
		swap_cache_del_folio(folio);
		goto redirty;
	}
	if (nr_pages > 1)
		goto try_split;
redirty:
	folio_mark_dirty(folio);
	return AOP_WRITEPAGE_ACTIVATE;	/* Return with folio locked */
}
EXPORT_SYMBOL_GPL(shmem_writeout);

#if defined(CONFIG_NUMA) && defined(CONFIG_TMPFS)
static void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
{
	char buffer[64];

	if (!mpol || mpol->mode == MPOL_DEFAULT)
		return;		/* show nothing */

	mpol_to_str(buffer, sizeof(buffer), mpol);

	seq_printf(seq, ",mpol=%s", buffer);
}

static struct mempolicy *shmem_get_sbmpol(struct shmem_sb_info *sbinfo)
{
	struct mempolicy *mpol = NULL;
	if (sbinfo->mpol) {
		raw_spin_lock(&sbinfo->stat_lock);	/* prevent replace/use races */
		mpol = sbinfo->mpol;
		mpol_get(mpol);
		raw_spin_unlock(&sbinfo->stat_lock);
	}
	return mpol;
}
#else /* !CONFIG_NUMA || !CONFIG_TMPFS */
static inline void shmem_show_mpol(struct seq_file *seq, struct mempolicy *mpol)
{
}
static inline struct mempolicy *shmem_get_sbmpol(struct shmem_sb_info *sbinfo)
{
	return NULL;
}
#endif /* CONFIG_NUMA && CONFIG_TMPFS */

static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info,
			pgoff_t index, unsigned int order, pgoff_t *ilx);

static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp,
			struct shmem_inode_info *info, pgoff_t index)
{
	struct mempolicy *mpol;
	pgoff_t ilx;
	struct folio *folio;

	mpol = shmem_get_pgoff_policy(info, index, 0, &ilx);
	folio = swap_cluster_readahead(swap, gfp, mpol, ilx);
	mpol_cond_put(mpol);

	return folio;
}

/*
 * Make sure huge_gfp is always more limited than limit_gfp.
 * Some of the flags set permissions, while others set limitations.
 */
static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
{
	gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
	gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
	gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
	gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);

	/* Allow allocations only from the originally specified zones. */
	result |= zoneflags;

	/*
	 * Minimize the result gfp by taking the union with the deny flags,
	 * and the intersection of the allow flags.
	 */
	result |= (limit_gfp & denyflags);
	result |= (huge_gfp & limit_gfp) & allowflags;

	return result;
}

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
bool shmem_hpage_pmd_enabled(void)
{
	if (shmem_huge == SHMEM_HUGE_DENY)
		return false;
	if (test_bit(HPAGE_PMD_ORDER, &huge_shmem_orders_always))
		return true;
	if (test_bit(HPAGE_PMD_ORDER, &huge_shmem_orders_madvise))
		return true;
	if (test_bit(HPAGE_PMD_ORDER, &huge_shmem_orders_within_size))
		return true;
	if (test_bit(HPAGE_PMD_ORDER, &huge_shmem_orders_inherit) &&
	    shmem_huge != SHMEM_HUGE_NEVER)
		return true;

	return false;
}

unsigned long shmem_allowable_huge_orders(struct inode *inode,
				struct vm_area_struct *vma, pgoff_t index,
				loff_t write_end, bool shmem_huge_force)
{
	unsigned long mask = READ_ONCE(huge_shmem_orders_always);
	unsigned long within_size_orders = READ_ONCE(huge_shmem_orders_within_size);
	vm_flags_t vm_flags = vma ? vma->vm_flags : 0;
	unsigned int global_orders;

	if (thp_disabled_by_hw() || (vma && vma_thp_disabled(vma, vm_flags, shmem_huge_force)))
		return 0;

	global_orders = shmem_huge_global_enabled(inode, index, write_end,
						  shmem_huge_force, vma, vm_flags);
	/* Tmpfs huge pages allocation */
	if (!vma || !vma_is_anon_shmem(vma))
		return global_orders;

	/*
	 * Following the 'deny' semantics of the top level, force the huge
	 * option off from all mounts.
	 */
	if (shmem_huge == SHMEM_HUGE_DENY)
		return 0;

	/*
	 * Only allow inherit orders if the top-level value is 'force', which
	 * means non-PMD sized THP can not override 'huge' mount option now.
	 */
	if (shmem_huge == SHMEM_HUGE_FORCE)
		return READ_ONCE(huge_shmem_orders_inherit);

	/* Allow mTHP that will be fully within i_size. */
	mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0);

	if (vm_flags & VM_HUGEPAGE)
		mask |= READ_ONCE(huge_shmem_orders_madvise);

	if (global_orders > 0)
		mask |= READ_ONCE(huge_shmem_orders_inherit);

	return THP_ORDERS_ALL_FILE_DEFAULT & mask;
}

static unsigned long shmem_suitable_orders(struct inode *inode, struct vm_fault *vmf,
					   struct address_space *mapping, pgoff_t index,
					   unsigned long orders)
{
	struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
	pgoff_t aligned_index;
	unsigned long pages;
	int order;

	if (vma) {
		orders = thp_vma_suitable_orders(vma, vmf->address, orders);
		if (!orders)
			return 0;
	}

	/* Find the highest order that can add into the page cache */
	order = highest_order(orders);
	while (orders) {
		pages = 1UL << order;
		aligned_index = round_down(index, pages);
		/*
		 * Check for conflict before waiting on a huge allocation.
		 * Conflict might be that a huge page has just been allocated
		 * and added to page cache by a racing thread, or that there
		 * is already at least one small page in the huge extent.
		 * Be careful to retry when appropriate, but not forever!
		 * Elsewhere -EEXIST would be the right code, but not here.
		 */
		if (!xa_find(&mapping->i_pages, &aligned_index,
			     aligned_index + pages - 1, XA_PRESENT))
			break;
		order = next_order(&orders, order);
	}

	return orders;
}
#else
static unsigned long shmem_suitable_orders(struct inode *inode, struct vm_fault *vmf,
					   struct address_space *mapping, pgoff_t index,
					   unsigned long orders)
{
	return 0;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */

static struct folio *shmem_alloc_folio(gfp_t gfp, int order,
		struct shmem_inode_info *info, pgoff_t index)
{
	struct mempolicy *mpol;
	pgoff_t ilx;
	struct folio *folio;

	mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
	folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
	mpol_cond_put(mpol);

	return folio;
}

static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
		gfp_t gfp, struct inode *inode, pgoff_t index,
		struct mm_struct *fault_mm, unsigned long orders)
{
	struct address_space *mapping = inode->i_mapping;
	struct shmem_inode_info *info = SHMEM_I(inode);
	unsigned long suitable_orders = 0;
	struct folio *folio = NULL;
	pgoff_t aligned_index;
	long pages;
	int error, order;

	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
		orders = 0;

	if (orders > 0) {
		suitable_orders = shmem_suitable_orders(inode, vmf,
							mapping, index, orders);

		order = highest_order(suitable_orders);
		while (suitable_orders) {
			pages = 1UL << order;
			aligned_index = round_down(index, pages);
			folio = shmem_alloc_folio(gfp, order, info, aligned_index);
			if (folio) {
				index = aligned_index;
				goto allocated;
			}

			if (pages == HPAGE_PMD_NR)
				count_vm_event(THP_FILE_FALLBACK);
			count_mthp_stat(order, MTHP_STAT_SHMEM_FALLBACK);
			order = next_order(&suitable_orders, order);
		}
	} else {
		pages = 1;
		folio = shmem_alloc_folio(gfp, 0, info, index);
	}
	if (!folio)
		return ERR_PTR(-ENOMEM);

allocated:
	__folio_set_locked(folio);
	__folio_set_swapbacked(folio);

	gfp &= GFP_RECLAIM_MASK;
	error = mem_cgroup_charge(folio, fault_mm, gfp);
	if (error) {
		if (xa_find(&mapping->i_pages, &index,
				index + pages - 1, XA_PRESENT)) {
			error = -EEXIST;
		} else if (pages > 1) {
			if (pages == HPAGE_PMD_NR) {
				count_vm_event(THP_FILE_FALLBACK);
				count_vm_event(THP_FILE_FALLBACK_CHARGE);
			}
			count_mthp_stat(folio_order(folio), MTHP_STAT_SHMEM_FALLBACK);
			count_mthp_stat(folio_order(folio), MTHP_STAT_SHMEM_FALLBACK_CHARGE);
		}
		goto unlock;
	}

	error = shmem_add_to_page_cache(folio, mapping, index, NULL, gfp);
	if (error)
		goto unlock;

	error = shmem_inode_acct_blocks(inode, pages);
	if (error) {
		struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
		long freed;
		/*
		 * Try to reclaim some space by splitting a few
		 * large folios beyond i_size on the filesystem.
		 */
		shmem_unused_huge_shrink(sbinfo, NULL, pages);
		/*
		 * And do a shmem_recalc_inode() to account for freed pages:
		 * except our folio is there in cache, so not quite balanced.
		 */
		spin_lock(&info->lock);
		freed = pages + info->alloced - info->swapped -
			READ_ONCE(mapping->nrpages);
		if (freed > 0)
			info->alloced -= freed;
		spin_unlock(&info->lock);
		if (freed > 0)
			shmem_inode_unacct_blocks(inode, freed);
		error = shmem_inode_acct_blocks(inode, pages);
		if (error) {
			filemap_remove_folio(folio);
			goto unlock;
		}
	}

	shmem_recalc_inode(inode, pages, 0);
	folio_add_lru(folio);
	return folio;

unlock:
	folio_unlock(folio);
	folio_put(folio);
	return ERR_PTR(error);
}

static struct folio *shmem_swap_alloc_folio(struct inode *inode,
		struct vm_area_struct *vma, pgoff_t index,
		swp_entry_t entry, int order, gfp_t gfp)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct folio *new, *swapcache;
	int nr_pages = 1 << order;
	gfp_t alloc_gfp;

	/*
	 * We have arrived here because our zones are constrained, so don't
	 * limit chance of success with further cpuset and node constraints.
	 */
	gfp &= ~GFP_CONSTRAINT_MASK;
	alloc_gfp = gfp;
	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
		if (WARN_ON_ONCE(order))
			return ERR_PTR(-EINVAL);
	} else if (order) {
		/*
		 * If uffd is active for the vma, we need per-page fault
		 * fidelity to maintain the uffd semantics, then fallback
		 * to swapin order-0 folio, as well as for zswap case.
		 * Any existing sub folio in the swap cache also blocks
		 * mTHP swapin.
		 */
		if ((vma && unlikely(userfaultfd_armed(vma))) ||
		     !zswap_never_enabled() ||
		     non_swapcache_batch(entry, nr_pages) != nr_pages)
			goto fallback;

		alloc_gfp = limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
	}
retry:
	new = shmem_alloc_folio(alloc_gfp, order, info, index);
	if (!new) {
		new = ERR_PTR(-ENOMEM);
		goto fallback;
	}

	if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,
					   alloc_gfp, entry)) {
		folio_put(new);
		new = ERR_PTR(-ENOMEM);
		goto fallback;
	}

	swapcache = swapin_folio(entry, new);
	if (swapcache != new) {
		folio_put(new);
		if (!swapcache) {
			/*
			 * The new folio is charged already, swapin can
			 * only fail due to another raced swapin.
			 */
			new = ERR_PTR(-EEXIST);
			goto fallback;
		}
	}
	return swapcache;
fallback:
	/* Order 0 swapin failed, nothing to fallback to, abort */
	if (!order)
		return new;
	entry.val += index - round_down(index, nr_pages);
	alloc_gfp = gfp;
	nr_pages = 1;
	order = 0;
	goto retry;
}

/*
 * When a page is moved from swapcache to shmem filecache (either by the
 * usual swapin of shmem_get_folio_gfp(), or by the less common swapoff of
 * shmem_unuse_inode()), it may have been read in earlier from swap, in
 * ignorance of the mapping it belongs to.  If that mapping has special
 * constraints (like the gma500 GEM driver, which requires RAM below 4GB),
 * we may need to copy to a suitable page before moving to filecache.
 *
 * In a future release, this may well be extended to respect cpuset and
 * NUMA mempolicy, and applied also to anonymous pages in do_swap_page();
 * but for now it is a simple matter of zone.
 */
static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp)
{
	return folio_zonenum(folio) > gfp_zone(gfp);
}

static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,
				struct shmem_inode_info *info, pgoff_t index,
				struct vm_area_struct *vma)
{
	struct swap_cluster_info *ci;
	struct folio *new, *old = *foliop;
	swp_entry_t entry = old->swap;
	int nr_pages = folio_nr_pages(old);
	int error = 0;

	/*
	 * We have arrived here because our zones are constrained, so don't
	 * limit chance of success by further cpuset and node constraints.
	 */
	gfp &= ~GFP_CONSTRAINT_MASK;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (nr_pages > 1) {
		gfp_t huge_gfp = vma_thp_gfp_mask(vma);

		gfp = limit_gfp_mask(huge_gfp, gfp);
	}
#endif

	new = shmem_alloc_folio(gfp, folio_order(old), info, index);
	if (!new)
		return -ENOMEM;

	folio_ref_add(new, nr_pages);
	folio_copy(new, old);
	flush_dcache_folio(new);

	__folio_set_locked(new);
	__folio_set_swapbacked(new);
	folio_mark_uptodate(new);
	new->swap = entry;
	folio_set_swapcache(new);

	ci = swap_cluster_get_and_lock_irq(old);
	__swap_cache_replace_folio(ci, old, new);
	mem_cgroup_replace_folio(old, new);
	shmem_update_stats(new, nr_pages);
	shmem_update_stats(old, -nr_pages);
	swap_cluster_unlock_irq(ci);

	folio_add_lru(new);
	*foliop = new;

	folio_clear_swapcache(old);
	old->private = NULL;

	folio_unlock(old);
	/*
	 * The old folio are removed from swap cache, drop the 'nr_pages'
	 * reference, as well as one temporary reference getting from swap
	 * cache.
	 */
	folio_put_refs(old, nr_pages + 1);
	return error;
}

static void shmem_set_folio_swapin_error(struct inode *inode, pgoff_t index,
					 struct folio *folio, swp_entry_t swap)
{
	struct address_space *mapping = inode->i_mapping;
	swp_entry_t swapin_error;
	void *old;
	int nr_pages;

	swapin_error = make_poisoned_swp_entry();
	old = xa_cmpxchg_irq(&mapping->i_pages, index,
			     swp_to_radix_entry(swap),
			     swp_to_radix_entry(swapin_error), 0);
	if (old != swp_to_radix_entry(swap))
		return;

	nr_pages = folio_nr_pages(folio);
	folio_wait_writeback(folio);
	folio_put_swap(folio, NULL);
	swap_cache_del_folio(folio);
	/*
	 * Don't treat swapin error folio as alloced. Otherwise inode->i_blocks
	 * won't be 0 when inode is released and thus trigger WARN_ON(i_blocks)
	 * in shmem_evict_inode().
	 */
	shmem_recalc_inode(inode, -nr_pages, -nr_pages);
}

static int shmem_split_large_entry(struct inode *inode, pgoff_t index,
				   swp_entry_t swap, gfp_t gfp)
{
	struct address_space *mapping = inode->i_mapping;
	XA_STATE_ORDER(xas, &mapping->i_pages, index, 0);
	int split_order = 0;
	int i;

	/* Convert user data gfp flags to xarray node gfp flags */
	gfp &= GFP_RECLAIM_MASK;

	for (;;) {
		void *old = NULL;
		int cur_order;
		pgoff_t swap_index;

		xas_lock_irq(&xas);
		old = xas_load(&xas);
		if (!xa_is_value(old) || swp_to_radix_entry(swap) != old) {
			xas_set_err(&xas, -EEXIST);
			goto unlock;
		}

		cur_order = xas_get_order(&xas);
		if (!cur_order)
			goto unlock;

		/* Try to split large swap entry in pagecache */
		swap_index = round_down(index, 1 << cur_order);
		split_order = xas_try_split_min_order(cur_order);

		while (cur_order > 0) {
			pgoff_t aligned_index =
				round_down(index, 1 << cur_order);
			pgoff_t swap_offset = aligned_index - swap_index;

			xas_set_order(&xas, index, split_order);
			xas_try_split(&xas, old, cur_order);
			if (xas_error(&xas))
				goto unlock;

			/*
			 * Re-set the swap entry after splitting, and the swap
			 * offset of the original large entry must be continuous.
			 */
			for (i = 0; i < 1 << cur_order;
			     i += (1 << split_order)) {
				swp_entry_t tmp;

				tmp = swp_entry(swp_type(swap),
						swp_offset(swap) + swap_offset +
							i);
				__xa_store(&mapping->i_pages, aligned_index + i,
					   swp_to_radix_entry(tmp), 0);
			}
			cur_order = split_order;
			split_order = xas_try_split_min_order(split_order);
		}

unlock:
		xas_unlock_irq(&xas);

		if (!xas_nomem(&xas, gfp))
			break;
	}

	if (xas_error(&xas))
		return xas_error(&xas);

	return 0;
}

/*
 * Swap in the folio pointed to by *foliop.
 * Caller has to make sure that *foliop contains a valid swapped folio.
 * Returns 0 and the folio in foliop if success. On failure, returns the
 * error code and NULL in *foliop.
 */
static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
			     struct folio **foliop, enum sgp_type sgp,
			     gfp_t gfp, struct vm_area_struct *vma,
			     vm_fault_t *fault_type)
{
	struct address_space *mapping = inode->i_mapping;
	struct mm_struct *fault_mm = vma ? vma->vm_mm : NULL;
	struct shmem_inode_info *info = SHMEM_I(inode);
	swp_entry_t swap;
	softleaf_t index_entry;
	struct swap_info_struct *si;
	struct folio *folio = NULL;
	int error, nr_pages, order;
	pgoff_t offset;

	VM_BUG_ON(!*foliop || !xa_is_value(*foliop));
	index_entry = radix_to_swp_entry(*foliop);
	swap = index_entry;
	*foliop = NULL;

	if (softleaf_is_poison_marker(index_entry))
		return -EIO;

	si = get_swap_device(index_entry);
	order = shmem_confirm_swap(mapping, index, index_entry);
	if (unlikely(!si)) {
		if (order < 0)
			return -EEXIST;
		else
			return -EINVAL;
	}
	if (unlikely(order < 0)) {
		put_swap_device(si);
		return -EEXIST;
	}

	/* index may point to the middle of a large entry, get the sub entry */
	if (order) {
		offset = index - round_down(index, 1 << order);
		swap = swp_entry(swp_type(swap), swp_offset(swap) + offset);
	}

	/* Look it up and read it in.. */
	folio = swap_cache_get_folio(swap);
	if (!folio) {
		if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
			/* Direct swapin skipping swap cache & readahead */
			folio = shmem_swap_alloc_folio(inode, vma, index,
						       index_entry, order, gfp);
			if (IS_ERR(folio)) {
				error = PTR_ERR(folio);
				folio = NULL;
				goto failed;
			}
		} else {
			/* Cached swapin only supports order 0 folio */
			folio = shmem_swapin_cluster(swap, gfp, info, index);
			if (!folio) {
				error = -ENOMEM;
				goto failed;
			}
		}
		if (fault_type) {
			*fault_type |= VM_FAULT_MAJOR;
			count_vm_event(PGMAJFAULT);
			count_memcg_event_mm(fault_mm, PGMAJFAULT);
		}
	} else {
		swap_update_readahead(folio, NULL, 0);
	}

	if (order > folio_order(folio)) {
		/*
		 * Swapin may get smaller folios due to various reasons:
		 * It may fallback to order 0 due to memory pressure or race,
		 * swap readahead may swap in order 0 folios into swapcache
		 * asynchronously, while the shmem mapping can still stores
		 * large swap entries. In such cases, we should split the
		 * large swap entry to prevent possible data corruption.
		 */
		error = shmem_split_large_entry(inode, index, index_entry, gfp);
		if (error)
			goto failed_nolock;
	}

	/*
	 * If the folio is large, round down swap and index by folio size.
	 * No matter what race occurs, the swap layer ensures we either get
	 * a valid folio that has its swap entry aligned by size, or a
	 * temporarily invalid one which we'll abort very soon and retry.
	 *
	 * shmem_add_to_page_cache ensures the whole range contains expected
	 * entries and prevents any corruption, so any race split is fine
	 * too, it will succeed as long as the entries are still there.
	 */
	nr_pages = folio_nr_pages(folio);
	if (nr_pages > 1) {
		swap.val = round_down(swap.val, nr_pages);
		index = round_down(index, nr_pages);
	}

	/*
	 * We have to do this with the folio locked to prevent races.
	 * The shmem_confirm_swap below only checks if the first swap
	 * entry matches the folio, that's enough to ensure the folio
	 * is not used outside of shmem, as shmem swap entries
	 * and swap cache folios are never partially freed.
	 */
	folio_lock(folio);
	if (!folio_matches_swap_entry(folio, swap) ||
	    shmem_confirm_swap(mapping, index, swap) < 0) {
		error = -EEXIST;
		goto unlock;
	}
	if (!folio_test_uptodate(folio)) {
		error = -EIO;
		goto failed;
	}
	folio_wait_writeback(folio);

	/*
	 * Some architectures may have to restore extra metadata to the
	 * folio after reading from swap.
	 */
	arch_swap_restore(folio_swap(swap, folio), folio);

	if (shmem_should_replace_folio(folio, gfp)) {
		error = shmem_replace_folio(&folio, gfp, info, index, vma);
		if (error)
			goto failed;
	}

	error = shmem_add_to_page_cache(folio, mapping, index,
					swp_to_radix_entry(swap), gfp);
	if (error)
		goto failed;

	shmem_recalc_inode(inode, 0, -nr_pages);

	if (sgp == SGP_WRITE)
		folio_mark_accessed(folio);

	folio_put_swap(folio, NULL);
	swap_cache_del_folio(folio);
	folio_mark_dirty(folio);
	put_swap_device(si);

	*foliop = folio;
	return 0;
failed:
	if (shmem_confirm_swap(mapping, index, swap) < 0)
		error = -EEXIST;
	if (error == -EIO)
		shmem_set_folio_swapin_error(inode, index, folio, swap);
unlock:
	if (folio)
		folio_unlock(folio);
failed_nolock:
	if (folio)
		folio_put(folio);
	put_swap_device(si);

	return error;
}

/*
 * shmem_get_folio_gfp - find page in cache, or get from swap, or allocate
 *
 * If we allocate a new one we do not mark it dirty. That's up to the
 * vm. If we swap it in we mark it dirty since we also free the swap
 * entry since a page cannot live in both the swap and page cache.
 *
 * vmf and fault_type are only supplied by shmem_fault: otherwise they are NULL.
 */
static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
		loff_t write_end, struct folio **foliop, enum sgp_type sgp,
		gfp_t gfp, struct vm_fault *vmf, vm_fault_t *fault_type)
{
	struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
	struct mm_struct *fault_mm;
	struct folio *folio;
	int error;
	bool alloced;
	unsigned long orders = 0;

	if (WARN_ON_ONCE(!shmem_mapping(inode->i_mapping)))
		return -EINVAL;

	if (index > (MAX_LFS_FILESIZE >> PAGE_SHIFT))
		return -EFBIG;
repeat:
	if (sgp <= SGP_CACHE &&
	    ((loff_t)index << PAGE_SHIFT) >= i_size_read(inode))
		return -EINVAL;

	alloced = false;
	fault_mm = vma ? vma->vm_mm : NULL;

	folio = filemap_get_entry(inode->i_mapping, index);
	if (folio && vma && userfaultfd_minor(vma)) {
		if (!xa_is_value(folio))
			folio_put(folio);
		*fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
		return 0;
	}

	if (xa_is_value(folio)) {
		error = shmem_swapin_folio(inode, index, &folio,
					   sgp, gfp, vma, fault_type);
		if (error == -EEXIST)
			goto repeat;

		*foliop = folio;
		return error;
	}

	if (folio) {
		folio_lock(folio);

		/* Has the folio been truncated or swapped out? */
		if (unlikely(folio->mapping != inode->i_mapping)) {
			folio_unlock(folio);
			folio_put(folio);
			goto repeat;
		}
		if (sgp == SGP_WRITE)
			folio_mark_accessed(folio);
		if (folio_test_uptodate(folio))
			goto out;
		/* fallocated folio */
		if (sgp != SGP_READ)
			goto clear;
		folio_unlock(folio);
		folio_put(folio);
	}

	/*
	 * SGP_READ: succeed on hole, with NULL folio, letting caller zero.
	 * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail.
	 */
	*foliop = NULL;
	if (sgp == SGP_READ)
		return 0;
	if (sgp == SGP_NOALLOC)
		return -ENOENT;

	/*
	 * Fast cache lookup and swap lookup did not find it: allocate.
	 */

	if (vma && userfaultfd_missing(vma)) {
		*fault_type = handle_userfault(vmf, VM_UFFD_MISSING);
		return 0;
	}

	/* Find hugepage orders that are allowed for anonymous shmem and tmpfs. */
	orders = shmem_allowable_huge_orders(inode, vma, index, write_end, false);
	if (orders > 0) {
		gfp_t huge_gfp;

		huge_gfp = vma_thp_gfp_mask(vma);
		huge_gfp = limit_gfp_mask(huge_gfp, gfp);
		folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
				inode, index, fault_mm, orders);
		if (!IS_ERR(folio)) {
			if (folio_test_pmd_mappable(folio))
				count_vm_event(THP_FILE_ALLOC);
			count_mthp_stat(folio_order(folio), MTHP_STAT_SHMEM_ALLOC);
			goto alloced;
		}
		if (PTR_ERR(folio) == -EEXIST)
			goto repeat;
	}

	folio = shmem_alloc_and_add_folio(vmf, gfp, inode, index, fault_mm, 0);
	if (IS_ERR(folio)) {
		error = PTR_ERR(folio);
		if (error == -EEXIST)
			goto repeat;
		folio = NULL;
		goto unlock;
	}

alloced:
	alloced = true;
	if (folio_test_large(folio) &&
	    DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) <
					folio_next_index(folio)) {
		struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
		struct shmem_inode_info *info = SHMEM_I(inode);
		/*
		 * Part of the large folio is beyond i_size: subject
		 * to shrink under memory pressure.
		 */
		spin_lock(&sbinfo->shrinklist_lock);
		/*
		 * _careful to defend against unlocked access to
		 * ->shrink_list in shmem_unused_huge_shrink()
		 */
		if (list_empty_careful(&info->shrinklist)) {
			list_add_tail(&info->shrinklist,
				      &sbinfo->shrinklist);
			sbinfo->shrinklist_len++;
		}
		spin_unlock(&sbinfo->shrinklist_lock);
	}

	if (sgp == SGP_WRITE)
		folio_set_referenced(folio);
	/*
	 * Let SGP_FALLOC use the SGP_WRITE optimization on a new folio.
	 */
	if (sgp == SGP_FALLOC)
		sgp = SGP_WRITE;
clear:
	/*
	 * Let SGP_WRITE caller clear ends if write does not fill folio;
	 * but SGP_FALLOC on a folio fallocated earlier must initialize
	 * it now, lest undo on failure cancel our earlier guarantee.
	 */
	if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) {
		long i, n = folio_nr_pages(folio);

		for (i = 0; i < n; i++)
			clear_highpage(folio_page(folio, i));
		flush_dcache_folio(folio);
		folio_mark_uptodate(folio);
	}

	/* Perhaps the file has been truncated since we checked */
	if (sgp <= SGP_CACHE &&
	    ((loff_t)index << PAGE_SHIFT) >= i_size_read(inode)) {
		error = -EINVAL;
		goto unlock;
	}
out:
	*foliop = folio;
	return 0;

	/*
	 * Error recovery.
	 */
unlock:
	if (alloced)
		filemap_remove_folio(folio);
	shmem_recalc_inode(inode, 0, 0);
	if (folio) {
		folio_unlock(folio);
		folio_put(folio);
	}
	return error;
}

/**
 * shmem_get_folio - find, and lock a shmem folio.
 * @inode:	inode to search
 * @index:	the page index.
 * @write_end:	end of a write, could extend inode size
 * @foliop:	pointer to the folio if found
 * @sgp:	SGP_* flags to control behavior
 *
 * Looks up the page cache entry at @inode & @index.  If a folio is
 * present, it is returned locked with an increased refcount.
 *
 * If the caller modifies data in the folio, it must call folio_mark_dirty()
 * before unlocking the folio to ensure that the folio is not reclaimed.
 * There is no need to reserve space before calling folio_mark_dirty().
 *
 * When no folio is found, the behavior depends on @sgp:
 *  - for SGP_READ, *@foliop is %NULL and 0 is returned
 *  - for SGP_NOALLOC, *@foliop is %NULL and -ENOENT is returned
 *  - for all other flags a new folio is allocated, inserted into the
 *    page cache and returned locked in @foliop.
 *
 * Context: May sleep.
 * Return: 0 if successful, else a negative error code.
 */
int shmem_get_folio(struct inode *inode, pgoff_t index, loff_t write_end,
		    struct folio **foliop, enum sgp_type sgp)
{
	return shmem_get_folio_gfp(inode, index, write_end, foliop, sgp,
			mapping_gfp_mask(inode->i_mapping), NULL, NULL);
}
EXPORT_SYMBOL_GPL(shmem_get_folio);

/*
 * This is like autoremove_wake_function, but it removes the wait queue
 * entry unconditionally - even if something else had already woken the
 * target.
 */
static int synchronous_wake_function(wait_queue_entry_t *wait,
			unsigned int mode, int sync, void *key)
{
	int ret = default_wake_function(wait, mode, sync, key);
	list_del_init(&wait->entry);
	return ret;
}

/*
 * Trinity finds that probing a hole which tmpfs is punching can
 * prevent the hole-punch from ever completing: which in turn
 * locks writers out with its hold on i_rwsem.  So refrain from
 * faulting pages into the hole while it's being punched.  Although
 * shmem_undo_range() does remove the additions, it may be unable to
 * keep up, as each new page needs its own unmap_mapping_range() call,
 * and the i_mmap tree grows ever slower to scan if new vmas are added.
 *
 * It does not matter if we sometimes reach this check just before the
 * hole-punch begins, so that one fault then races with the punch:
 * we just need to make racing faults a rare case.
 *
 * The implementation below would be much simpler if we just used a
 * standard mutex or completion: but we cannot take i_rwsem in fault,
 * and bloating every shmem inode for this unlikely case would be sad.
 */
static vm_fault_t shmem_falloc_wait(struct vm_fault *vmf, struct inode *inode)
{
	struct shmem_falloc *shmem_falloc;
	struct file *fpin = NULL;
	vm_fault_t ret = 0;

	spin_lock(&inode->i_lock);
	shmem_falloc = inode->i_private;
	if (shmem_falloc &&
	    shmem_falloc->waitq &&
	    vmf->pgoff >= shmem_falloc->start &&
	    vmf->pgoff < shmem_falloc->next) {
		wait_queue_head_t *shmem_falloc_waitq;
		DEFINE_WAIT_FUNC(shmem_fault_wait, synchronous_wake_function);

		ret = VM_FAULT_NOPAGE;
		fpin = maybe_unlock_mmap_for_io(vmf, NULL);
		shmem_falloc_waitq = shmem_falloc->waitq;
		prepare_to_wait(shmem_falloc_waitq, &shmem_fault_wait,
				TASK_UNINTERRUPTIBLE);
		spin_unlock(&inode->i_lock);
		schedule();

		/*
		 * shmem_falloc_waitq points into the shmem_fallocate()
		 * stack of the hole-punching task: shmem_falloc_waitq
		 * is usually invalid by the time we reach here, but
		 * finish_wait() does not dereference it in that case;
		 * though i_lock needed lest racing with wake_up_all().
		 */
		spin_lock(&inode->i_lock);
		finish_wait(shmem_falloc_waitq, &shmem_fault_wait);
	}
	spin_unlock(&inode->i_lock);
	if (fpin) {
		fput(fpin);
		ret = VM_FAULT_RETRY;
	}
	return ret;
}

static vm_fault_t shmem_fault(struct vm_fault *vmf)
{
	struct inode *inode = file_inode(vmf->vma->vm_file);
	gfp_t gfp = mapping_gfp_mask(inode->i_mapping);
	struct folio *folio = NULL;
	vm_fault_t ret = 0;
	int err;

	/*
	 * Trinity finds that probing a hole which tmpfs is punching can
	 * prevent the hole-punch from ever completing: noted in i_private.
	 */
	if (unlikely(inode->i_private)) {
		ret = shmem_falloc_wait(vmf, inode);
		if (ret)
			return ret;
	}

	WARN_ON_ONCE(vmf->page != NULL);
	err = shmem_get_folio_gfp(inode, vmf->pgoff, 0, &folio, SGP_CACHE,
				  gfp, vmf, &ret);
	if (err)
		return vmf_error(err);
	if (folio) {
		vmf->page = folio_file_page(folio, vmf->pgoff);
		ret |= VM_FAULT_LOCKED;
	}
	return ret;
}

unsigned long shmem_get_unmapped_area(struct file *file,
				      unsigned long uaddr, unsigned long len,
				      unsigned long pgoff, unsigned long flags)
{
	unsigned long addr;
	unsigned long offset;
	unsigned long inflated_len;
	unsigned long inflated_addr;
	unsigned long inflated_offset;
	unsigned long hpage_size;

	if (len > TASK_SIZE)
		return -ENOMEM;

	addr = mm_get_unmapped_area(file, uaddr, len, pgoff, flags);

	if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE))
		return addr;
	if (IS_ERR_VALUE(addr))
		return addr;
	if (addr & ~PAGE_MASK)
		return addr;
	if (addr > TASK_SIZE - len)
		return addr;

	if (shmem_huge == SHMEM_HUGE_DENY)
		return addr;
	if (flags & MAP_FIXED)
		return addr;
	/*
	 * Our priority is to support MAP_SHARED mapped hugely;
	 * and support MAP_PRIVATE mapped hugely too, until it is COWed.
	 * But if caller specified an address hint and we allocated area there
	 * successfully, respect that as before.
	 */
	if (uaddr == addr)
		return addr;

	hpage_size = HPAGE_PMD_SIZE;
	if (shmem_huge != SHMEM_HUGE_FORCE) {
		struct super_block *sb;
		unsigned long __maybe_unused hpage_orders;
		int order = 0;

		if (file) {
			VM_BUG_ON(file->f_op != &shmem_file_operations);
			sb = file_inode(file)->i_sb;
		} else {
			/*
			 * Called directly from mm/mmap.c, or drivers/char/mem.c
			 * for "/dev/zero", to create a shared anonymous object.
			 */
			if (IS_ERR(shm_mnt))
				return addr;
			sb = shm_mnt->mnt_sb;

			/*
			 * Find the highest mTHP order used for anonymous shmem to
			 * provide a suitable alignment address.
			 */
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
			hpage_orders = READ_ONCE(huge_shmem_orders_always);
			hpage_orders |= READ_ONCE(huge_shmem_orders_within_size);
			hpage_orders |= READ_ONCE(huge_shmem_orders_madvise);
			if (SHMEM_SB(sb)->huge != SHMEM_HUGE_NEVER)
				hpage_orders |= READ_ONCE(huge_shmem_orders_inherit);

			if (hpage_orders > 0) {
				order = highest_order(hpage_orders);
				hpage_size = PAGE_SIZE << order;
			}
#endif
		}
		if (SHMEM_SB(sb)->huge == SHMEM_HUGE_NEVER && !order)
			return addr;
	}

	if (len < hpage_size)
		return addr;

	offset = (pgoff << PAGE_SHIFT) & (hpage_size - 1);
	if (offset && offset + len < 2 * hpage_size)
		return addr;
	if ((addr & (hpage_size - 1)) == offset)
		return addr;

	inflated_len = len + hpage_size - PAGE_SIZE;
	if (inflated_len > TASK_SIZE)
		return addr;
	if (inflated_len < len)
		return addr;

	inflated_addr = mm_get_unmapped_area(NULL, uaddr, inflated_len, 0, flags);
	if (IS_ERR_VALUE(inflated_addr))
		return addr;
	if (inflated_addr & ~PAGE_MASK)
		return addr;

	inflated_offset = inflated_addr & (hpage_size - 1);
	inflated_addr += offset - inflated_offset;
	if (inflated_offset > offset)
		inflated_addr += hpage_size;

	if (inflated_addr > TASK_SIZE - len)
		return addr;
	return inflated_addr;
}

#ifdef CONFIG_NUMA
static int shmem_set_policy(struct vm_area_struct *vma, struct mempolicy *mpol)
{
	struct inode *inode = file_inode(vma->vm_file);
	return mpol_set_shared_policy(&SHMEM_I(inode)->policy, vma, mpol);
}

static struct mempolicy *shmem_get_policy(struct vm_area_struct *vma,
					  unsigned long addr, pgoff_t *ilx)
{
	struct inode *inode = file_inode(vma->vm_file);
	pgoff_t index;

	/*
	 * Bias interleave by inode number to distribute better across nodes;
	 * but this interface is independent of which page order is used, so
	 * supplies only that bias, letting caller apply the offset (adjusted
	 * by page order, as in shmem_get_pgoff_policy() and get_vma_policy()).
	 */
	*ilx = inode->i_ino;
	index = ((addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff;
	return mpol_shared_policy_lookup(&SHMEM_I(inode)->policy, index);
}

static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info,
			pgoff_t index, unsigned int order, pgoff_t *ilx)
{
	struct mempolicy *mpol;

	/* Bias interleave by inode number to distribute better across nodes */
	*ilx = info->vfs_inode.i_ino + (index >> order);

	mpol = mpol_shared_policy_lookup(&info->policy, index);
	return mpol ? mpol : get_task_policy(current);
}
#else
static struct mempolicy *shmem_get_pgoff_policy(struct shmem_inode_info *info,
			pgoff_t index, unsigned int order, pgoff_t *ilx)
{
	*ilx = 0;
	return NULL;
}
#endif /* CONFIG_NUMA */

int shmem_lock(struct file *file, int lock, struct ucounts *ucounts)
{
	struct inode *inode = file_inode(file);
	struct shmem_inode_info *info = SHMEM_I(inode);
	int retval = -ENOMEM;

	/*
	 * What serializes the accesses to info->flags?
	 * ipc_lock_object() when called from shmctl_do_lock(),
	 * no serialization needed when called from shm_destroy().
	 */
	if (lock && !(info->flags & SHMEM_F_LOCKED)) {
		if (!user_shm_lock(inode->i_size, ucounts))
			goto out_nomem;
		info->flags |= SHMEM_F_LOCKED;
		mapping_set_unevictable(file->f_mapping);
	}
	if (!lock && (info->flags & SHMEM_F_LOCKED) && ucounts) {
		user_shm_unlock(inode->i_size, ucounts);
		info->flags &= ~SHMEM_F_LOCKED;
		mapping_clear_unevictable(file->f_mapping);
	}
	retval = 0;

out_nomem:
	return retval;
}

static int shmem_mmap_prepare(struct vm_area_desc *desc)
{
	struct file *file = desc->file;
	struct inode *inode = file_inode(file);

	file_accessed(file);
	/* This is anonymous shared memory if it is unlinked at the time of mmap */
	if (inode->i_nlink)
		desc->vm_ops = &shmem_vm_ops;
	else
		desc->vm_ops = &shmem_anon_vm_ops;
	return 0;
}

static int shmem_file_open(struct inode *inode, struct file *file)
{
	file->f_mode |= FMODE_CAN_ODIRECT;
	return generic_file_open(inode, file);
}

#ifdef CONFIG_TMPFS_XATTR
static int shmem_initxattrs(struct inode *, const struct xattr *, void *);

#if IS_ENABLED(CONFIG_UNICODE)
/*
 * shmem_inode_casefold_flags - Deal with casefold file attribute flag
 *
 * The casefold file attribute needs some special checks. I can just be added to
 * an empty dir, and can't be removed from a non-empty dir.
 */
static int shmem_inode_casefold_flags(struct inode *inode, unsigned int fsflags,
				      struct dentry *dentry, unsigned int *i_flags)
{
	unsigned int old = inode->i_flags;
	struct super_block *sb = inode->i_sb;

	if (fsflags & FS_CASEFOLD_FL) {
		if (!(old & S_CASEFOLD)) {
			if (!sb->s_encoding)
				return -EOPNOTSUPP;

			if (!S_ISDIR(inode->i_mode))
				return -ENOTDIR;

			if (dentry && !simple_empty(dentry))
				return -ENOTEMPTY;
		}

		*i_flags = *i_flags | S_CASEFOLD;
	} else if (old & S_CASEFOLD) {
		if (dentry && !simple_empty(dentry))
			return -ENOTEMPTY;
	}

	return 0;
}
#else
static int shmem_inode_casefold_flags(struct inode *inode, unsigned int fsflags,
				      struct dentry *dentry, unsigned int *i_flags)
{
	if (fsflags & FS_CASEFOLD_FL)
		return -EOPNOTSUPP;

	return 0;
}
#endif

/*
 * chattr's fsflags are unrelated to extended attributes,
 * but tmpfs has chosen to enable them under the same config option.
 */
static int shmem_set_inode_flags(struct inode *inode, unsigned int fsflags, struct dentry *dentry)
{
	unsigned int i_flags = 0;
	int ret;

	ret = shmem_inode_casefold_flags(inode, fsflags, dentry, &i_flags);
	if (ret)
		return ret;

	if (fsflags & FS_NOATIME_FL)
		i_flags |= S_NOATIME;
	if (fsflags & FS_APPEND_FL)
		i_flags |= S_APPEND;
	if (fsflags & FS_IMMUTABLE_FL)
		i_flags |= S_IMMUTABLE;
	/*
	 * But FS_NODUMP_FL does not require any action in i_flags.
	 */
	inode_set_flags(inode, i_flags, S_NOATIME | S_APPEND | S_IMMUTABLE | S_CASEFOLD);

	return 0;
}
#else
static void shmem_set_inode_flags(struct inode *inode, unsigned int fsflags, struct dentry *dentry)
{
}
#define shmem_initxattrs NULL
#endif

static struct offset_ctx *shmem_get_offset_ctx(struct inode *inode)
{
	return &SHMEM_I(inode)->dir_offsets;
}

static struct inode *__shmem_get_inode(struct mnt_idmap *idmap,
				       struct super_block *sb,
				       struct inode *dir, umode_t mode,
				       dev_t dev, vma_flags_t flags)
{
	struct inode *inode;
	struct shmem_inode_info *info;
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);
	ino_t ino;
	int err;

	err = shmem_reserve_inode(sb, &ino);
	if (err)
		return ERR_PTR(err);

	inode = new_inode(sb);
	if (!inode) {
		shmem_free_inode(sb, 0);
		return ERR_PTR(-ENOSPC);
	}

	inode->i_ino = ino;
	inode_init_owner(idmap, inode, dir, mode);
	inode->i_blocks = 0;
	simple_inode_init_ts(inode);
	inode->i_generation = get_random_u32();
	info = SHMEM_I(inode);
	memset(info, 0, (char *)inode - (char *)info);
	spin_lock_init(&info->lock);
	atomic_set(&info->stop_eviction, 0);
	info->seals = F_SEAL_SEAL;
	info->flags = vma_flags_test(&flags, VMA_NORESERVE_BIT)
		? SHMEM_F_NORESERVE : 0;
	info->i_crtime = inode_get_mtime(inode);
	info->fsflags = (dir == NULL) ? 0 :
		SHMEM_I(dir)->fsflags & SHMEM_FL_INHERITED;
	if (info->fsflags)
		shmem_set_inode_flags(inode, info->fsflags, NULL);
	INIT_LIST_HEAD(&info->shrinklist);
	INIT_LIST_HEAD(&info->swaplist);
	simple_xattrs_init(&info->xattrs);
	cache_no_acl(inode);
	if (sbinfo->noswap)
		mapping_set_unevictable(inode->i_mapping);

	/* Don't consider 'deny' for emergencies and 'force' for testing */
	if (sbinfo->huge)
		mapping_set_large_folios(inode->i_mapping);

	switch (mode & S_IFMT) {
	default:
		inode->i_op = &shmem_special_inode_operations;
		init_special_inode(inode, mode, dev);
		break;
	case S_IFREG:
		inode->i_mapping->a_ops = &shmem_aops;
		inode->i_op = &shmem_inode_operations;
		inode->i_fop = &shmem_file_operations;
		mpol_shared_policy_init(&info->policy,
					 shmem_get_sbmpol(sbinfo));
		break;
	case S_IFDIR:
		inc_nlink(inode);
		/* Some things misbehave if size == 0 on a directory */
		inode->i_size = 2 * BOGO_DIRENT_SIZE;
		inode->i_op = &shmem_dir_inode_operations;
		inode->i_fop = &simple_offset_dir_operations;
		simple_offset_init(shmem_get_offset_ctx(inode));
		break;
	case S_IFLNK:
		/*
		 * Must not load anything in the rbtree,
		 * mpol_free_shared_policy will not be called.
		 */
		mpol_shared_policy_init(&info->policy, NULL);
		break;
	}

	lockdep_annotate_inode_mutex_key(inode);
	return inode;
}

#ifdef CONFIG_TMPFS_QUOTA
static struct inode *shmem_get_inode(struct mnt_idmap *idmap,
				     struct super_block *sb, struct inode *dir,
				     umode_t mode, dev_t dev, vma_flags_t flags)
{
	int err;
	struct inode *inode;

	inode = __shmem_get_inode(idmap, sb, dir, mode, dev, flags);
	if (IS_ERR(inode))
		return inode;

	err = dquot_initialize(inode);
	if (err)
		goto errout;

	err = dquot_alloc_inode(inode);
	if (err) {
		dquot_drop(inode);
		goto errout;
	}
	return inode;

errout:
	inode->i_flags |= S_NOQUOTA;
	iput(inode);
	return ERR_PTR(err);
}
#else
static struct inode *shmem_get_inode(struct mnt_idmap *idmap,
				     struct super_block *sb, struct inode *dir,
				     umode_t mode, dev_t dev, vma_flags_t flags)
{
	return __shmem_get_inode(idmap, sb, dir, mode, dev, flags);
}
#endif /* CONFIG_TMPFS_QUOTA */

#ifdef CONFIG_USERFAULTFD
int shmem_mfill_atomic_pte(pmd_t *dst_pmd,
			   struct vm_area_struct *dst_vma,
			   unsigned long dst_addr,
			   unsigned long src_addr,
			   uffd_flags_t flags,
			   struct folio **foliop)
{
	struct inode *inode = file_inode(dst_vma->vm_file);
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct address_space *mapping = inode->i_mapping;
	gfp_t gfp = mapping_gfp_mask(mapping);
	pgoff_t pgoff = linear_page_index(dst_vma, dst_addr);
	void *page_kaddr;
	struct folio *folio;
	int ret;
	pgoff_t max_off;

	if (shmem_inode_acct_blocks(inode, 1)) {
		/*
		 * We may have got a page, returned -ENOENT triggering a retry,
		 * and now we find ourselves with -ENOMEM. Release the page, to
		 * avoid a BUG_ON in our caller.
		 */
		if (unlikely(*foliop)) {
			folio_put(*foliop);
			*foliop = NULL;
		}
		return -ENOMEM;
	}

	if (!*foliop) {
		ret = -ENOMEM;
		folio = shmem_alloc_folio(gfp, 0, info, pgoff);
		if (!folio)
			goto out_unacct_blocks;

		if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY)) {
			page_kaddr = kmap_local_folio(folio, 0);
			/*
			 * The read mmap_lock is held here.  Despite the
			 * mmap_lock being read recursive a deadlock is still
			 * possible if a writer has taken a lock.  For example:
			 *
			 * process A thread 1 takes read lock on own mmap_lock
			 * process A thread 2 calls mmap, blocks taking write lock
			 * process B thread 1 takes page fault, read lock on own mmap lock
			 * process B thread 2 calls mmap, blocks taking write lock
			 * process A thread 1 blocks taking read lock on process B
			 * process B thread 1 blocks taking read lock on process A
			 *
			 * Disable page faults to prevent potential deadlock
			 * and retry the copy outside the mmap_lock.
			 */
			pagefault_disable();
			ret = copy_from_user(page_kaddr,
					     (const void __user *)src_addr,
					     PAGE_SIZE);
			pagefault_enable();
			kunmap_local(page_kaddr);

			/* fallback to copy_from_user outside mmap_lock */
			if (unlikely(ret)) {
				*foliop = folio;
				ret = -ENOENT;
				/* don't free the page */
				goto out_unacct_blocks;
			}

			flush_dcache_folio(folio);
		} else {		/* ZEROPAGE */
			clear_user_highpage(&folio->page, dst_addr);
		}
	} else {
		folio = *foliop;
		VM_BUG_ON_FOLIO(folio_test_large(folio), folio);
		*foliop = NULL;
	}

	VM_BUG_ON(folio_test_locked(folio));
	VM_BUG_ON(folio_test_swapbacked(folio));
	__folio_set_locked(folio);
	__folio_set_swapbacked(folio);
	__folio_mark_uptodate(folio);

	ret = -EFAULT;
	max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE);
	if (unlikely(pgoff >= max_off))
		goto out_release;

	ret = mem_cgroup_charge(folio, dst_vma->vm_mm, gfp);
	if (ret)
		goto out_release;
	ret = shmem_add_to_page_cache(folio, mapping, pgoff, NULL, gfp);
	if (ret)
		goto out_release;

	ret = mfill_atomic_install_pte(dst_pmd, dst_vma, dst_addr,
				       &folio->page, true, flags);
	if (ret)
		goto out_delete_from_cache;

	shmem_recalc_inode(inode, 1, 0);
	folio_unlock(folio);
	return 0;
out_delete_from_cache:
	filemap_remove_folio(folio);
out_release:
	folio_unlock(folio);
	folio_put(folio);
out_unacct_blocks:
	shmem_inode_unacct_blocks(inode, 1);
	return ret;
}
#endif /* CONFIG_USERFAULTFD */

#ifdef CONFIG_TMPFS
static const struct inode_operations shmem_symlink_inode_operations;
static const struct inode_operations shmem_short_symlink_operations;

static int
shmem_write_begin(const struct kiocb *iocb, struct address_space *mapping,
		  loff_t pos, unsigned len,
		  struct folio **foliop, void **fsdata)
{
	struct inode *inode = mapping->host;
	struct shmem_inode_info *info = SHMEM_I(inode);
	pgoff_t index = pos >> PAGE_SHIFT;
	struct folio *folio;
	int ret = 0;

	/* i_rwsem is held by caller */
	if (unlikely(info->seals & (F_SEAL_GROW |
				   F_SEAL_WRITE | F_SEAL_FUTURE_WRITE))) {
		if (info->seals & (F_SEAL_WRITE | F_SEAL_FUTURE_WRITE))
			return -EPERM;
		if ((info->seals & F_SEAL_GROW) && pos + len > inode->i_size)
			return -EPERM;
	}

	if (unlikely((info->flags & SHMEM_F_MAPPING_FROZEN) &&
		     pos + len > inode->i_size))
		return -EPERM;

	ret = shmem_get_folio(inode, index, pos + len, &folio, SGP_WRITE);
	if (ret)
		return ret;

	if (folio_contain_hwpoisoned_page(folio)) {
		folio_unlock(folio);
		folio_put(folio);
		return -EIO;
	}

	*foliop = folio;
	return 0;
}

static int
shmem_write_end(const struct kiocb *iocb, struct address_space *mapping,
		loff_t pos, unsigned len, unsigned copied,
		struct folio *folio, void *fsdata)
{
	struct inode *inode = mapping->host;

	if (pos + copied > inode->i_size)
		i_size_write(inode, pos + copied);

	if (!folio_test_uptodate(folio)) {
		if (copied < folio_size(folio)) {
			size_t from = offset_in_folio(folio, pos);
			folio_zero_segments(folio, 0, from,
					from + copied, folio_size(folio));
		}
		folio_mark_uptodate(folio);
	}
	folio_mark_dirty(folio);
	folio_unlock(folio);
	folio_put(folio);

	return copied;
}

static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)
{
	struct file *file = iocb->ki_filp;
	struct inode *inode = file_inode(file);
	struct address_space *mapping = inode->i_mapping;
	pgoff_t index;
	unsigned long offset;
	int error = 0;
	ssize_t retval = 0;

	for (;;) {
		struct folio *folio = NULL;
		struct page *page = NULL;
		unsigned long nr, ret;
		loff_t end_offset, i_size = i_size_read(inode);
		bool fallback_page_copy = false;
		size_t fsize;

		if (unlikely(iocb->ki_pos >= i_size))
			break;

		index = iocb->ki_pos >> PAGE_SHIFT;
		error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
		if (error) {
			if (error == -EINVAL)
				error = 0;
			break;
		}
		if (folio) {
			folio_unlock(folio);

			page = folio_file_page(folio, index);
			if (PageHWPoison(page)) {
				folio_put(folio);
				error = -EIO;
				break;
			}

			if (folio_test_large(folio) &&
			    folio_test_has_hwpoisoned(folio))
				fallback_page_copy = true;
		}

		/*
		 * We must evaluate after, since reads (unlike writes)
		 * are called without i_rwsem protection against truncate
		 */
		i_size = i_size_read(inode);
		if (unlikely(iocb->ki_pos >= i_size)) {
			if (folio)
				folio_put(folio);
			break;
		}
		end_offset = min_t(loff_t, i_size, iocb->ki_pos + to->count);
		if (folio && likely(!fallback_page_copy))
			fsize = folio_size(folio);
		else
			fsize = PAGE_SIZE;
		offset = iocb->ki_pos & (fsize - 1);
		nr = min_t(loff_t, end_offset - iocb->ki_pos, fsize - offset);

		if (folio) {
			/*
			 * If users can be writing to this page using arbitrary
			 * virtual addresses, take care about potential aliasing
			 * before reading the page on the kernel side.
			 */
			if (mapping_writably_mapped(mapping)) {
				if (likely(!fallback_page_copy))
					flush_dcache_folio(folio);
				else
					flush_dcache_page(page);
			}

			/*
			 * Mark the folio accessed if we read the beginning.
			 */
			if (!offset)
				folio_mark_accessed(folio);
			/*
			 * Ok, we have the page, and it's up-to-date, so
			 * now we can copy it to user space...
			 */
			if (likely(!fallback_page_copy))
				ret = copy_folio_to_iter(folio, offset, nr, to);
			else
				ret = copy_page_to_iter(page, offset, nr, to);
			folio_put(folio);
		} else if (user_backed_iter(to)) {
			/*
			 * Copy to user tends to be so well optimized, but
			 * clear_user() not so much, that it is noticeably
			 * faster to copy the zero page instead of clearing.
			 */
			ret = copy_page_to_iter(ZERO_PAGE(0), offset, nr, to);
		} else {
			/*
			 * But submitting the same page twice in a row to
			 * splice() - or others? - can result in confusion:
			 * so don't attempt that optimization on pipes etc.
			 */
			ret = iov_iter_zero(nr, to);
		}

		retval += ret;
		iocb->ki_pos += ret;

		if (!iov_iter_count(to))
			break;
		if (ret < nr) {
			error = -EFAULT;
			break;
		}
		cond_resched();
	}

	file_accessed(file);
	return retval ? retval : error;
}

static ssize_t shmem_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
	struct file *file = iocb->ki_filp;
	struct inode *inode = file->f_mapping->host;
	ssize_t ret;

	inode_lock(inode);
	ret = generic_write_checks(iocb, from);
	if (ret <= 0)
		goto unlock;
	ret = file_remove_privs(file);
	if (ret)
		goto unlock;
	ret = file_update_time(file);
	if (ret)
		goto unlock;
	ret = generic_perform_write(iocb, from);
unlock:
	inode_unlock(inode);
	return ret;
}

static bool zero_pipe_buf_get(struct pipe_inode_info *pipe,
			      struct pipe_buffer *buf)
{
	return true;
}

static void zero_pipe_buf_release(struct pipe_inode_info *pipe,
				  struct pipe_buffer *buf)
{
}

static bool zero_pipe_buf_try_steal(struct pipe_inode_info *pipe,
				    struct pipe_buffer *buf)
{
	return false;
}

static const struct pipe_buf_operations zero_pipe_buf_ops = {
	.release	= zero_pipe_buf_release,
	.try_steal	= zero_pipe_buf_try_steal,
	.get		= zero_pipe_buf_get,
};

static size_t splice_zeropage_into_pipe(struct pipe_inode_info *pipe,
					loff_t fpos, size_t size)
{
	size_t offset = fpos & ~PAGE_MASK;

	size = min_t(size_t, size, PAGE_SIZE - offset);

	if (!pipe_is_full(pipe)) {
		struct pipe_buffer *buf = pipe_head_buf(pipe);

		*buf = (struct pipe_buffer) {
			.ops	= &zero_pipe_buf_ops,
			.page	= ZERO_PAGE(0),
			.offset	= offset,
			.len	= size,
		};
		pipe->head++;
	}

	return size;
}

static ssize_t shmem_file_splice_read(struct file *in, loff_t *ppos,
				      struct pipe_inode_info *pipe,
				      size_t len, unsigned int flags)
{
	struct inode *inode = file_inode(in);
	struct address_space *mapping = inode->i_mapping;
	struct folio *folio = NULL;
	size_t total_spliced = 0, used, npages, n, part;
	loff_t isize;
	int error = 0;

	/* Work out how much data we can actually add into the pipe */
	used = pipe_buf_usage(pipe);
	npages = max_t(ssize_t, pipe->max_usage - used, 0);
	len = min_t(size_t, len, npages * PAGE_SIZE);

	do {
		bool fallback_page_splice = false;
		struct page *page = NULL;
		pgoff_t index;
		size_t size;

		if (*ppos >= i_size_read(inode))
			break;

		index = *ppos >> PAGE_SHIFT;
		error = shmem_get_folio(inode, index, 0, &folio, SGP_READ);
		if (error) {
			if (error == -EINVAL)
				error = 0;
			break;
		}
		if (folio) {
			folio_unlock(folio);

			page = folio_file_page(folio, index);
			if (PageHWPoison(page)) {
				error = -EIO;
				break;
			}

			if (folio_test_large(folio) &&
			    folio_test_has_hwpoisoned(folio))
				fallback_page_splice = true;
		}

		/*
		 * i_size must be checked after we know the pages are Uptodate.
		 *
		 * Checking i_size after the check allows us to calculate
		 * the correct value for "nr", which means the zero-filled
		 * part of the page is not copied back to userspace (unless
		 * another truncate extends the file - this is desired though).
		 */
		isize = i_size_read(inode);
		if (unlikely(*ppos >= isize))
			break;
		/*
		 * Fallback to PAGE_SIZE splice if the large folio has hwpoisoned
		 * pages.
		 */
		size = len;
		if (unlikely(fallback_page_splice)) {
			size_t offset = *ppos & ~PAGE_MASK;

			size = umin(size, PAGE_SIZE - offset);
		}
		part = min_t(loff_t, isize - *ppos, size);

		if (folio) {
			/*
			 * If users can be writing to this page using arbitrary
			 * virtual addresses, take care about potential aliasing
			 * before reading the page on the kernel side.
			 */
			if (mapping_writably_mapped(mapping)) {
				if (likely(!fallback_page_splice))
					flush_dcache_folio(folio);
				else
					flush_dcache_page(page);
			}
			folio_mark_accessed(folio);
			/*
			 * Ok, we have the page, and it's up-to-date, so we can
			 * now splice it into the pipe.
			 */
			n = splice_folio_into_pipe(pipe, folio, *ppos, part);
			folio_put(folio);
			folio = NULL;
		} else {
			n = splice_zeropage_into_pipe(pipe, *ppos, part);
		}

		if (!n)
			break;
		len -= n;
		total_spliced += n;
		*ppos += n;
		in->f_ra.prev_pos = *ppos;
		if (pipe_is_full(pipe))
			break;

		cond_resched();
	} while (len);

	if (folio)
		folio_put(folio);

	file_accessed(in);
	return total_spliced ? total_spliced : error;
}

static loff_t shmem_file_llseek(struct file *file, loff_t offset, int whence)
{
	struct address_space *mapping = file->f_mapping;
	struct inode *inode = mapping->host;

	if (whence != SEEK_DATA && whence != SEEK_HOLE)
		return generic_file_llseek_size(file, offset, whence,
					MAX_LFS_FILESIZE, i_size_read(inode));
	if (offset < 0)
		return -ENXIO;

	inode_lock(inode);
	/* We're holding i_rwsem so we can access i_size directly */
	offset = mapping_seek_hole_data(mapping, offset, inode->i_size, whence);
	if (offset >= 0)
		offset = vfs_setpos(file, offset, MAX_LFS_FILESIZE);
	inode_unlock(inode);
	return offset;
}

static long shmem_fallocate(struct file *file, int mode, loff_t offset,
							 loff_t len)
{
	struct inode *inode = file_inode(file);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_falloc shmem_falloc;
	pgoff_t start, index, end, undo_fallocend;
	int error;

	if (mode & ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE))
		return -EOPNOTSUPP;

	inode_lock(inode);

	if (info->flags & SHMEM_F_MAPPING_FROZEN) {
		error = -EPERM;
		goto out;
	}

	if (mode & FALLOC_FL_PUNCH_HOLE) {
		struct address_space *mapping = file->f_mapping;
		loff_t unmap_start = round_up(offset, PAGE_SIZE);
		loff_t unmap_end = round_down(offset + len, PAGE_SIZE) - 1;
		DECLARE_WAIT_QUEUE_HEAD_ONSTACK(shmem_falloc_waitq);

		/* protected by i_rwsem */
		if (info->seals & (F_SEAL_WRITE | F_SEAL_FUTURE_WRITE)) {
			error = -EPERM;
			goto out;
		}

		shmem_falloc.waitq = &shmem_falloc_waitq;
		shmem_falloc.start = (u64)unmap_start >> PAGE_SHIFT;
		shmem_falloc.next = (unmap_end + 1) >> PAGE_SHIFT;
		spin_lock(&inode->i_lock);
		inode->i_private = &shmem_falloc;
		spin_unlock(&inode->i_lock);

		if ((u64)unmap_end > (u64)unmap_start)
			unmap_mapping_range(mapping, unmap_start,
					    1 + unmap_end - unmap_start, 0);
		shmem_truncate_range(inode, offset, offset + len - 1);
		/* No need to unmap again: hole-punching leaves COWed pages */

		spin_lock(&inode->i_lock);
		inode->i_private = NULL;
		wake_up_all(&shmem_falloc_waitq);
		WARN_ON_ONCE(!list_empty(&shmem_falloc_waitq.head));
		spin_unlock(&inode->i_lock);
		error = 0;
		goto out;
	}

	/* We need to check rlimit even when FALLOC_FL_KEEP_SIZE */
	error = inode_newsize_ok(inode, offset + len);
	if (error)
		goto out;

	if ((info->seals & F_SEAL_GROW) && offset + len > inode->i_size) {
		error = -EPERM;
		goto out;
	}

	start = offset >> PAGE_SHIFT;
	end = (offset + len + PAGE_SIZE - 1) >> PAGE_SHIFT;
	/* Try to avoid a swapstorm if len is impossible to satisfy */
	if (sbinfo->max_blocks && end - start > sbinfo->max_blocks) {
		error = -ENOSPC;
		goto out;
	}

	shmem_falloc.waitq = NULL;
	shmem_falloc.start = start;
	shmem_falloc.next  = start;
	shmem_falloc.nr_falloced = 0;
	shmem_falloc.nr_unswapped = 0;
	spin_lock(&inode->i_lock);
	inode->i_private = &shmem_falloc;
	spin_unlock(&inode->i_lock);

	/*
	 * info->fallocend is only relevant when huge pages might be
	 * involved: to prevent split_huge_page() freeing fallocated
	 * pages when FALLOC_FL_KEEP_SIZE committed beyond i_size.
	 */
	undo_fallocend = info->fallocend;
	if (info->fallocend < end)
		info->fallocend = end;

	for (index = start; index < end; ) {
		struct folio *folio;

		/*
		 * Check for fatal signal so that we abort early in OOM
		 * situations. We don't want to abort in case of non-fatal
		 * signals as large fallocate can take noticeable time and
		 * e.g. periodic timers may result in fallocate constantly
		 * restarting.
		 */
		if (fatal_signal_pending(current))
			error = -EINTR;
		else if (shmem_falloc.nr_unswapped > shmem_falloc.nr_falloced)
			error = -ENOMEM;
		else
			error = shmem_get_folio(inode, index, offset + len,
						&folio, SGP_FALLOC);
		if (error) {
			info->fallocend = undo_fallocend;
			/* Remove the !uptodate folios we added */
			if (index > start) {
				shmem_undo_range(inode,
				    (loff_t)start << PAGE_SHIFT,
				    ((loff_t)index << PAGE_SHIFT) - 1, true);
			}
			goto undone;
		}

		/*
		 * Here is a more important optimization than it appears:
		 * a second SGP_FALLOC on the same large folio will clear it,
		 * making it uptodate and un-undoable if we fail later.
		 */
		index = folio_next_index(folio);
		/* Beware 32-bit wraparound */
		if (!index)
			index--;

		/*
		 * Inform shmem_writeout() how far we have reached.
		 * No need for lock or barrier: we have the page lock.
		 */
		if (!folio_test_uptodate(folio))
			shmem_falloc.nr_falloced += index - shmem_falloc.next;
		shmem_falloc.next = index;

		/*
		 * If !uptodate, leave it that way so that freeable folios
		 * can be recognized if we need to rollback on error later.
		 * But mark it dirty so that memory pressure will swap rather
		 * than free the folios we are allocating (and SGP_CACHE folios
		 * might still be clean: we now need to mark those dirty too).
		 */
		folio_mark_dirty(folio);
		folio_unlock(folio);
		folio_put(folio);
		cond_resched();
	}

	if (!(mode & FALLOC_FL_KEEP_SIZE) && offset + len > inode->i_size)
		i_size_write(inode, offset + len);
undone:
	spin_lock(&inode->i_lock);
	inode->i_private = NULL;
	spin_unlock(&inode->i_lock);
out:
	if (!error)
		file_modified(file);
	inode_unlock(inode);
	return error;
}

static int shmem_statfs(struct dentry *dentry, struct kstatfs *buf)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(dentry->d_sb);

	buf->f_type = TMPFS_MAGIC;
	buf->f_bsize = PAGE_SIZE;
	buf->f_namelen = NAME_MAX;
	if (sbinfo->max_blocks) {
		buf->f_blocks = sbinfo->max_blocks;
		buf->f_bavail =
		buf->f_bfree  = sbinfo->max_blocks -
				percpu_counter_sum(&sbinfo->used_blocks);
	}
	if (sbinfo->max_inodes) {
		buf->f_files = sbinfo->max_inodes;
		buf->f_ffree = sbinfo->free_ispace / BOGO_INODE_SIZE;
	}
	/* else leave those fields 0 like simple_statfs */

	buf->f_fsid = uuid_to_fsid(dentry->d_sb->s_uuid.b);

	return 0;
}

/*
 * File creation. Allocate an inode, and we're done..
 */
static int
shmem_mknod(struct mnt_idmap *idmap, struct inode *dir,
	    struct dentry *dentry, umode_t mode, dev_t dev)
{
	struct inode *inode;
	int error;

	if (!generic_ci_validate_strict_name(dir, &dentry->d_name))
		return -EINVAL;

	inode = shmem_get_inode(idmap, dir->i_sb, dir, mode, dev,
				mk_vma_flags(VMA_NORESERVE_BIT));
	if (IS_ERR(inode))
		return PTR_ERR(inode);

	error = simple_acl_create(dir, inode);
	if (error)
		goto out_iput;
	error = security_inode_init_security(inode, dir, &dentry->d_name,
					     shmem_initxattrs, NULL);
	if (error && error != -EOPNOTSUPP)
		goto out_iput;

	error = simple_offset_add(shmem_get_offset_ctx(dir), dentry);
	if (error)
		goto out_iput;

	dir->i_size += BOGO_DIRENT_SIZE;
	inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
	inode_inc_iversion(dir);

	d_make_persistent(dentry, inode);
	return error;

out_iput:
	iput(inode);
	return error;
}

static int
shmem_tmpfile(struct mnt_idmap *idmap, struct inode *dir,
	      struct file *file, umode_t mode)
{
	struct inode *inode;
	int error;

	inode = shmem_get_inode(idmap, dir->i_sb, dir, mode, 0,
				mk_vma_flags(VMA_NORESERVE_BIT));
	if (IS_ERR(inode)) {
		error = PTR_ERR(inode);
		goto err_out;
	}
	error = security_inode_init_security(inode, dir, NULL,
					     shmem_initxattrs, NULL);
	if (error && error != -EOPNOTSUPP)
		goto out_iput;
	error = simple_acl_create(dir, inode);
	if (error)
		goto out_iput;
	d_tmpfile(file, inode);

err_out:
	return finish_open_simple(file, error);
out_iput:
	iput(inode);
	return error;
}

static struct dentry *shmem_mkdir(struct mnt_idmap *idmap, struct inode *dir,
				  struct dentry *dentry, umode_t mode)
{
	int error;

	error = shmem_mknod(idmap, dir, dentry, mode | S_IFDIR, 0);
	if (error)
		return ERR_PTR(error);
	inc_nlink(dir);
	return NULL;
}

static int shmem_create(struct mnt_idmap *idmap, struct inode *dir,
			struct dentry *dentry, umode_t mode, bool excl)
{
	return shmem_mknod(idmap, dir, dentry, mode | S_IFREG, 0);
}

/*
 * Link a file..
 */
static int shmem_link(struct dentry *old_dentry, struct inode *dir,
		      struct dentry *dentry)
{
	struct inode *inode = d_inode(old_dentry);
	int ret;

	/*
	 * No ordinary (disk based) filesystem counts links as inodes;
	 * but each new link needs a new dentry, pinning lowmem, and
	 * tmpfs dentries cannot be pruned until they are unlinked.
	 * But if an O_TMPFILE file is linked into the tmpfs, the
	 * first link must skip that, to get the accounting right.
	 */
	if (inode->i_nlink) {
		ret = shmem_reserve_inode(inode->i_sb, NULL);
		if (ret)
			return ret;
	}

	ret = simple_offset_add(shmem_get_offset_ctx(dir), dentry);
	if (ret) {
		if (inode->i_nlink)
			shmem_free_inode(inode->i_sb, 0);
		return ret;
	}

	dir->i_size += BOGO_DIRENT_SIZE;
	inode_inc_iversion(dir);
	return simple_link(old_dentry, dir, dentry);
}

static int shmem_unlink(struct inode *dir, struct dentry *dentry)
{
	struct inode *inode = d_inode(dentry);

	if (inode->i_nlink > 1 && !S_ISDIR(inode->i_mode))
		shmem_free_inode(inode->i_sb, 0);

	simple_offset_remove(shmem_get_offset_ctx(dir), dentry);

	dir->i_size -= BOGO_DIRENT_SIZE;
	inode_inc_iversion(dir);
	simple_unlink(dir, dentry);

	/*
	 * For now, VFS can't deal with case-insensitive negative dentries, so
	 * we invalidate them
	 */
	if (IS_ENABLED(CONFIG_UNICODE) && IS_CASEFOLDED(dir))
		d_invalidate(dentry);

	return 0;
}

static int shmem_rmdir(struct inode *dir, struct dentry *dentry)
{
	if (!simple_empty(dentry))
		return -ENOTEMPTY;

	drop_nlink(d_inode(dentry));
	drop_nlink(dir);
	return shmem_unlink(dir, dentry);
}

static int shmem_whiteout(struct mnt_idmap *idmap,
			  struct inode *old_dir, struct dentry *old_dentry)
{
	struct dentry *whiteout;
	int error;

	whiteout = d_alloc(old_dentry->d_parent, &old_dentry->d_name);
	if (!whiteout)
		return -ENOMEM;
	error = shmem_mknod(idmap, old_dir, whiteout,
			    S_IFCHR | WHITEOUT_MODE, WHITEOUT_DEV);
	dput(whiteout);
	return error;
}

/*
 * The VFS layer already does all the dentry stuff for rename,
 * we just have to decrement the usage count for the target if
 * it exists so that the VFS layer correctly free's it when it
 * gets overwritten.
 */
static int shmem_rename2(struct mnt_idmap *idmap,
			 struct inode *old_dir, struct dentry *old_dentry,
			 struct inode *new_dir, struct dentry *new_dentry,
			 unsigned int flags)
{
	struct inode *inode = d_inode(old_dentry);
	int they_are_dirs = S_ISDIR(inode->i_mode);
	bool had_offset = false;
	int error;

	if (flags & ~(RENAME_NOREPLACE | RENAME_EXCHANGE | RENAME_WHITEOUT))
		return -EINVAL;

	if (flags & RENAME_EXCHANGE)
		return simple_offset_rename_exchange(old_dir, old_dentry,
						     new_dir, new_dentry);

	if (!simple_empty(new_dentry))
		return -ENOTEMPTY;

	error = simple_offset_add(shmem_get_offset_ctx(new_dir), new_dentry);
	if (error == -EBUSY)
		had_offset = true;
	else if (unlikely(error))
		return error;

	if (flags & RENAME_WHITEOUT) {
		error = shmem_whiteout(idmap, old_dir, old_dentry);
		if (error) {
			if (!had_offset)
				simple_offset_remove(shmem_get_offset_ctx(new_dir),
						     new_dentry);
			return error;
		}
	}

	simple_offset_rename(old_dir, old_dentry, new_dir, new_dentry);
	if (d_really_is_positive(new_dentry)) {
		(void) shmem_unlink(new_dir, new_dentry);
		if (they_are_dirs) {
			drop_nlink(d_inode(new_dentry));
			drop_nlink(old_dir);
		}
	} else if (they_are_dirs) {
		drop_nlink(old_dir);
		inc_nlink(new_dir);
	}

	old_dir->i_size -= BOGO_DIRENT_SIZE;
	new_dir->i_size += BOGO_DIRENT_SIZE;
	simple_rename_timestamp(old_dir, old_dentry, new_dir, new_dentry);
	inode_inc_iversion(old_dir);
	inode_inc_iversion(new_dir);
	return 0;
}

static int shmem_symlink(struct mnt_idmap *idmap, struct inode *dir,
			 struct dentry *dentry, const char *symname)
{
	int error;
	int len;
	struct inode *inode;
	struct folio *folio;
	char *link;

	len = strlen(symname) + 1;
	if (len > PAGE_SIZE)
		return -ENAMETOOLONG;

	inode = shmem_get_inode(idmap, dir->i_sb, dir, S_IFLNK | 0777, 0,
				mk_vma_flags(VMA_NORESERVE_BIT));
	if (IS_ERR(inode))
		return PTR_ERR(inode);

	error = security_inode_init_security(inode, dir, &dentry->d_name,
					     shmem_initxattrs, NULL);
	if (error && error != -EOPNOTSUPP)
		goto out_iput;

	error = simple_offset_add(shmem_get_offset_ctx(dir), dentry);
	if (error)
		goto out_iput;

	inode->i_size = len-1;
	if (len <= SHORT_SYMLINK_LEN) {
		link = kmemdup(symname, len, GFP_KERNEL);
		if (!link) {
			error = -ENOMEM;
			goto out_remove_offset;
		}
		inode->i_op = &shmem_short_symlink_operations;
		inode_set_cached_link(inode, link, len - 1);
	} else {
		inode_nohighmem(inode);
		inode->i_mapping->a_ops = &shmem_aops;
		error = shmem_get_folio(inode, 0, 0, &folio, SGP_WRITE);
		if (error)
			goto out_remove_offset;
		inode->i_op = &shmem_symlink_inode_operations;
		memcpy(folio_address(folio), symname, len);
		folio_mark_uptodate(folio);
		folio_mark_dirty(folio);
		folio_unlock(folio);
		folio_put(folio);
	}
	dir->i_size += BOGO_DIRENT_SIZE;
	inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir));
	inode_inc_iversion(dir);
	d_make_persistent(dentry, inode);
	return 0;

out_remove_offset:
	simple_offset_remove(shmem_get_offset_ctx(dir), dentry);
out_iput:
	iput(inode);
	return error;
}

static void shmem_put_link(void *arg)
{
	folio_mark_accessed(arg);
	folio_put(arg);
}

static const char *shmem_get_link(struct dentry *dentry, struct inode *inode,
				  struct delayed_call *done)
{
	struct folio *folio = NULL;
	int error;

	if (!dentry) {
		folio = filemap_get_folio(inode->i_mapping, 0);
		if (IS_ERR(folio))
			return ERR_PTR(-ECHILD);
		if (PageHWPoison(folio_page(folio, 0)) ||
		    !folio_test_uptodate(folio)) {
			folio_put(folio);
			return ERR_PTR(-ECHILD);
		}
	} else {
		error = shmem_get_folio(inode, 0, 0, &folio, SGP_READ);
		if (error)
			return ERR_PTR(error);
		if (!folio)
			return ERR_PTR(-ECHILD);
		if (PageHWPoison(folio_page(folio, 0))) {
			folio_unlock(folio);
			folio_put(folio);
			return ERR_PTR(-ECHILD);
		}
		folio_unlock(folio);
	}
	set_delayed_call(done, shmem_put_link, folio);
	return folio_address(folio);
}

#ifdef CONFIG_TMPFS_XATTR

static int shmem_fileattr_get(struct dentry *dentry, struct file_kattr *fa)
{
	struct shmem_inode_info *info = SHMEM_I(d_inode(dentry));

	fileattr_fill_flags(fa, info->fsflags & SHMEM_FL_USER_VISIBLE);

	return 0;
}

static int shmem_fileattr_set(struct mnt_idmap *idmap,
			      struct dentry *dentry, struct file_kattr *fa)
{
	struct inode *inode = d_inode(dentry);
	struct shmem_inode_info *info = SHMEM_I(inode);
	int ret, flags;

	if (fileattr_has_fsx(fa))
		return -EOPNOTSUPP;
	if (fa->flags & ~SHMEM_FL_USER_MODIFIABLE)
		return -EOPNOTSUPP;

	flags = (info->fsflags & ~SHMEM_FL_USER_MODIFIABLE) |
		(fa->flags & SHMEM_FL_USER_MODIFIABLE);

	ret = shmem_set_inode_flags(inode, flags, dentry);

	if (ret)
		return ret;

	info->fsflags = flags;

	inode_set_ctime_current(inode);
	inode_inc_iversion(inode);
	return 0;
}

/*
 * Superblocks without xattr inode operations may get some security.* xattr
 * support from the LSM "for free". As soon as we have any other xattrs
 * like ACLs, we also need to implement the security.* handlers at
 * filesystem level, though.
 */

/*
 * Callback for security_inode_init_security() for acquiring xattrs.
 */
static int shmem_initxattrs(struct inode *inode,
			    const struct xattr *xattr_array, void *fs_info)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	const struct xattr *xattr;
	struct simple_xattr *new_xattr;
	size_t ispace = 0;
	size_t len;

	if (sbinfo->max_inodes) {
		for (xattr = xattr_array; xattr->name != NULL; xattr++) {
			ispace += simple_xattr_space(xattr->name,
				xattr->value_len + XATTR_SECURITY_PREFIX_LEN);
		}
		if (ispace) {
			raw_spin_lock(&sbinfo->stat_lock);
			if (sbinfo->free_ispace < ispace)
				ispace = 0;
			else
				sbinfo->free_ispace -= ispace;
			raw_spin_unlock(&sbinfo->stat_lock);
			if (!ispace)
				return -ENOSPC;
		}
	}

	for (xattr = xattr_array; xattr->name != NULL; xattr++) {
		new_xattr = simple_xattr_alloc(xattr->value, xattr->value_len);
		if (!new_xattr)
			break;

		len = strlen(xattr->name) + 1;
		new_xattr->name = kmalloc(XATTR_SECURITY_PREFIX_LEN + len,
					  GFP_KERNEL_ACCOUNT);
		if (!new_xattr->name) {
			kvfree(new_xattr);
			break;
		}

		memcpy(new_xattr->name, XATTR_SECURITY_PREFIX,
		       XATTR_SECURITY_PREFIX_LEN);
		memcpy(new_xattr->name + XATTR_SECURITY_PREFIX_LEN,
		       xattr->name, len);

		simple_xattr_add(&info->xattrs, new_xattr);
	}

	if (xattr->name != NULL) {
		if (ispace) {
			raw_spin_lock(&sbinfo->stat_lock);
			sbinfo->free_ispace += ispace;
			raw_spin_unlock(&sbinfo->stat_lock);
		}
		simple_xattrs_free(&info->xattrs, NULL);
		return -ENOMEM;
	}

	return 0;
}

static int shmem_xattr_handler_get(const struct xattr_handler *handler,
				   struct dentry *unused, struct inode *inode,
				   const char *name, void *buffer, size_t size)
{
	struct shmem_inode_info *info = SHMEM_I(inode);

	name = xattr_full_name(handler, name);
	return simple_xattr_get(&info->xattrs, name, buffer, size);
}

static int shmem_xattr_handler_set(const struct xattr_handler *handler,
				   struct mnt_idmap *idmap,
				   struct dentry *unused, struct inode *inode,
				   const char *name, const void *value,
				   size_t size, int flags)
{
	struct shmem_inode_info *info = SHMEM_I(inode);
	struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
	struct simple_xattr *old_xattr;
	size_t ispace = 0;

	name = xattr_full_name(handler, name);
	if (value && sbinfo->max_inodes) {
		ispace = simple_xattr_space(name, size);
		raw_spin_lock(&sbinfo->stat_lock);
		if (sbinfo->free_ispace < ispace)
			ispace = 0;
		else
			sbinfo->free_ispace -= ispace;
		raw_spin_unlock(&sbinfo->stat_lock);
		if (!ispace)
			return -ENOSPC;
	}

	old_xattr = simple_xattr_set(&info->xattrs, name, value, size, flags);
	if (!IS_ERR(old_xattr)) {
		ispace = 0;
		if (old_xattr && sbinfo->max_inodes)
			ispace = simple_xattr_space(old_xattr->name,
						    old_xattr->size);
		simple_xattr_free(old_xattr);
		old_xattr = NULL;
		inode_set_ctime_current(inode);
		inode_inc_iversion(inode);
	}
	if (ispace) {
		raw_spin_lock(&sbinfo->stat_lock);
		sbinfo->free_ispace += ispace;
		raw_spin_unlock(&sbinfo->stat_lock);
	}
	return PTR_ERR(old_xattr);
}

static const struct xattr_handler shmem_security_xattr_handler = {
	.prefix = XATTR_SECURITY_PREFIX,
	.get = shmem_xattr_handler_get,
	.set = shmem_xattr_handler_set,
};

static const struct xattr_handler shmem_trusted_xattr_handler = {
	.prefix = XATTR_TRUSTED_PREFIX,
	.get = shmem_xattr_handler_get,
	.set = shmem_xattr_handler_set,
};

static const struct xattr_handler shmem_user_xattr_handler = {
	.prefix = XATTR_USER_PREFIX,
	.get = shmem_xattr_handler_get,
	.set = shmem_xattr_handler_set,
};

static const struct xattr_handler * const shmem_xattr_handlers[] = {
	&shmem_security_xattr_handler,
	&shmem_trusted_xattr_handler,
	&shmem_user_xattr_handler,
	NULL
};

static ssize_t shmem_listxattr(struct dentry *dentry, char *buffer, size_t size)
{
	struct shmem_inode_info *info = SHMEM_I(d_inode(dentry));
	return simple_xattr_list(d_inode(dentry), &info->xattrs, buffer, size);
}
#endif /* CONFIG_TMPFS_XATTR */

static const struct inode_operations shmem_short_symlink_operations = {
	.getattr	= shmem_getattr,
	.setattr	= shmem_setattr,
	.get_link	= simple_get_link,
#ifdef CONFIG_TMPFS_XATTR
	.listxattr	= shmem_listxattr,
#endif
};

static const struct inode_operations shmem_symlink_inode_operations = {
	.getattr	= shmem_getattr,
	.setattr	= shmem_setattr,
	.get_link	= shmem_get_link,
#ifdef CONFIG_TMPFS_XATTR
	.listxattr	= shmem_listxattr,
#endif
};

static struct dentry *shmem_get_parent(struct dentry *child)
{
	return ERR_PTR(-ESTALE);
}

static int shmem_match(struct inode *ino, void *vfh)
{
	__u32 *fh = vfh;
	__u64 inum = fh[2];
	inum = (inum << 32) | fh[1];
	return ino->i_ino == inum && fh[0] == ino->i_generation;
}

/* Find any alias of inode, but prefer a hashed alias */
static struct dentry *shmem_find_alias(struct inode *inode)
{
	struct dentry *alias = d_find_alias(inode);

	return alias ?: d_find_any_alias(inode);
}

static struct dentry *shmem_fh_to_dentry(struct super_block *sb,
		struct fid *fid, int fh_len, int fh_type)
{
	struct inode *inode;
	struct dentry *dentry = NULL;
	u64 inum;

	if (fh_len < 3)
		return NULL;

	inum = fid->raw[2];
	inum = (inum << 32) | fid->raw[1];

	inode = ilookup5(sb, (unsigned long)(inum + fid->raw[0]),
			shmem_match, fid->raw);
	if (inode) {
		dentry = shmem_find_alias(inode);
		iput(inode);
	}

	return dentry;
}

static int shmem_encode_fh(struct inode *inode, __u32 *fh, int *len,
				struct inode *parent)
{
	if (*len < 3) {
		*len = 3;
		return FILEID_INVALID;
	}

	if (inode_unhashed(inode)) {
		/* Unfortunately insert_inode_hash is not idempotent,
		 * so as we hash inodes here rather than at creation
		 * time, we need a lock to ensure we only try
		 * to do it once
		 */
		static DEFINE_SPINLOCK(lock);
		spin_lock(&lock);
		if (inode_unhashed(inode))
			__insert_inode_hash(inode,
					    inode->i_ino + inode->i_generation);
		spin_unlock(&lock);
	}

	fh[0] = inode->i_generation;
	fh[1] = inode->i_ino;
	fh[2] = ((__u64)inode->i_ino) >> 32;

	*len = 3;
	return 1;
}

static const struct export_operations shmem_export_ops = {
	.get_parent     = shmem_get_parent,
	.encode_fh      = shmem_encode_fh,
	.fh_to_dentry	= shmem_fh_to_dentry,
};

enum shmem_param {
	Opt_gid,
	Opt_huge,
	Opt_mode,
	Opt_mpol,
	Opt_nr_blocks,
	Opt_nr_inodes,
	Opt_size,
	Opt_uid,
	Opt_inode32,
	Opt_inode64,
	Opt_noswap,
	Opt_quota,
	Opt_usrquota,
	Opt_grpquota,
	Opt_usrquota_block_hardlimit,
	Opt_usrquota_inode_hardlimit,
	Opt_grpquota_block_hardlimit,
	Opt_grpquota_inode_hardlimit,
	Opt_casefold_version,
	Opt_casefold,
	Opt_strict_encoding,
};

static const struct constant_table shmem_param_enums_huge[] = {
	{"never",	SHMEM_HUGE_NEVER },
	{"always",	SHMEM_HUGE_ALWAYS },
	{"within_size",	SHMEM_HUGE_WITHIN_SIZE },
	{"advise",	SHMEM_HUGE_ADVISE },
	{}
};

const struct fs_parameter_spec shmem_fs_parameters[] = {
	fsparam_gid   ("gid",		Opt_gid),
	fsparam_enum  ("huge",		Opt_huge,  shmem_param_enums_huge),
	fsparam_u32oct("mode",		Opt_mode),
	fsparam_string("mpol",		Opt_mpol),
	fsparam_string("nr_blocks",	Opt_nr_blocks),
	fsparam_string("nr_inodes",	Opt_nr_inodes),
	fsparam_string("size",		Opt_size),
	fsparam_uid   ("uid",		Opt_uid),
	fsparam_flag  ("inode32",	Opt_inode32),
	fsparam_flag  ("inode64",	Opt_inode64),
	fsparam_flag  ("noswap",	Opt_noswap),
#ifdef CONFIG_TMPFS_QUOTA
	fsparam_flag  ("quota",		Opt_quota),
	fsparam_flag  ("usrquota",	Opt_usrquota),
	fsparam_flag  ("grpquota",	Opt_grpquota),
	fsparam_string("usrquota_block_hardlimit", Opt_usrquota_block_hardlimit),
	fsparam_string("usrquota_inode_hardlimit", Opt_usrquota_inode_hardlimit),
	fsparam_string("grpquota_block_hardlimit", Opt_grpquota_block_hardlimit),
	fsparam_string("grpquota_inode_hardlimit", Opt_grpquota_inode_hardlimit),
#endif
	fsparam_string("casefold",	Opt_casefold_version),
	fsparam_flag  ("casefold",	Opt_casefold),
	fsparam_flag  ("strict_encoding", Opt_strict_encoding),
	{}
};

#if IS_ENABLED(CONFIG_UNICODE)
static int shmem_parse_opt_casefold(struct fs_context *fc, struct fs_parameter *param,
				    bool latest_version)
{
	struct shmem_options *ctx = fc->fs_private;
	int version = UTF8_LATEST;
	struct unicode_map *encoding;
	char *version_str = param->string + 5;

	if (!latest_version) {
		if (strncmp(param->string, "utf8-", 5))
			return invalfc(fc, "Only UTF-8 encodings are supported "
				       "in the format: utf8-<version number>");

		version = utf8_parse_version(version_str);
		if (version < 0)
			return invalfc(fc, "Invalid UTF-8 version: %s", version_str);
	}

	encoding = utf8_load(version);

	if (IS_ERR(encoding)) {
		return invalfc(fc, "Failed loading UTF-8 version: utf8-%u.%u.%u\n",
			       unicode_major(version), unicode_minor(version),
			       unicode_rev(version));
	}

	pr_info("tmpfs: Using encoding : utf8-%u.%u.%u\n",
		unicode_major(version), unicode_minor(version), unicode_rev(version));

	ctx->encoding = encoding;

	return 0;
}
#else
static int shmem_parse_opt_casefold(struct fs_context *fc, struct fs_parameter *param,
				    bool latest_version)
{
	return invalfc(fc, "tmpfs: Kernel not built with CONFIG_UNICODE\n");
}
#endif

static int shmem_parse_one(struct fs_context *fc, struct fs_parameter *param)
{
	struct shmem_options *ctx = fc->fs_private;
	struct fs_parse_result result;
	unsigned long long size;
	char *rest;
	int opt;
	kuid_t kuid;
	kgid_t kgid;

	opt = fs_parse(fc, shmem_fs_parameters, param, &result);
	if (opt < 0)
		return opt;

	switch (opt) {
	case Opt_size:
		size = memparse(param->string, &rest);
		if (*rest == '%') {
			size <<= PAGE_SHIFT;
			size *= totalram_pages();
			do_div(size, 100);
			rest++;
		}
		if (*rest)
			goto bad_value;
		ctx->blocks = DIV_ROUND_UP(size, PAGE_SIZE);
		ctx->seen |= SHMEM_SEEN_BLOCKS;
		break;
	case Opt_nr_blocks:
		ctx->blocks = memparse(param->string, &rest);
		if (*rest || ctx->blocks > LONG_MAX)
			goto bad_value;
		ctx->seen |= SHMEM_SEEN_BLOCKS;
		break;
	case Opt_nr_inodes:
		ctx->inodes = memparse(param->string, &rest);
		if (*rest || ctx->inodes > ULONG_MAX / BOGO_INODE_SIZE)
			goto bad_value;
		ctx->seen |= SHMEM_SEEN_INODES;
		break;
	case Opt_mode:
		ctx->mode = result.uint_32 & 07777;
		break;
	case Opt_uid:
		kuid = result.uid;

		/*
		 * The requested uid must be representable in the
		 * filesystem's idmapping.
		 */
		if (!kuid_has_mapping(fc->user_ns, kuid))
			goto bad_value;

		ctx->uid = kuid;
		break;
	case Opt_gid:
		kgid = result.gid;

		/*
		 * The requested gid must be representable in the
		 * filesystem's idmapping.
		 */
		if (!kgid_has_mapping(fc->user_ns, kgid))
			goto bad_value;

		ctx->gid = kgid;
		break;
	case Opt_huge:
		ctx->huge = result.uint_32;
		if (ctx->huge != SHMEM_HUGE_NEVER &&
		    !(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
		      has_transparent_hugepage()))
			goto unsupported_parameter;
		ctx->seen |= SHMEM_SEEN_HUGE;
		break;
	case Opt_mpol:
		if (IS_ENABLED(CONFIG_NUMA)) {
			mpol_put(ctx->mpol);
			ctx->mpol = NULL;
			if (mpol_parse_str(param->string, &ctx->mpol))
				goto bad_value;
			break;
		}
		goto unsupported_parameter;
	case Opt_inode32:
		ctx->full_inums = false;
		ctx->seen |= SHMEM_SEEN_INUMS;
		break;
	case Opt_inode64:
		if (sizeof(ino_t) < 8) {
			return invalfc(fc,
				       "Cannot use inode64 with <64bit inums in kernel\n");
		}
		ctx->full_inums = true;
		ctx->seen |= SHMEM_SEEN_INUMS;
		break;
	case Opt_noswap:
		if ((fc->user_ns != &init_user_ns) || !capable(CAP_SYS_ADMIN)) {
			return invalfc(fc,
				       "Turning off swap in unprivileged tmpfs mounts unsupported");
		}
		ctx->noswap = true;
		break;
	case Opt_quota:
		if (fc->user_ns != &init_user_ns)
			return invalfc(fc, "Quotas in unprivileged tmpfs mounts are unsupported");
		ctx->seen |= SHMEM_SEEN_QUOTA;
		ctx->quota_types |= (QTYPE_MASK_USR | QTYPE_MASK_GRP);
		break;
	case Opt_usrquota:
		if (fc->user_ns != &init_user_ns)
			return invalfc(fc, "Quotas in unprivileged tmpfs mounts are unsupported");
		ctx->seen |= SHMEM_SEEN_QUOTA;
		ctx->quota_types |= QTYPE_MASK_USR;
		break;
	case Opt_grpquota:
		if (fc->user_ns != &init_user_ns)
			return invalfc(fc, "Quotas in unprivileged tmpfs mounts are unsupported");
		ctx->seen |= SHMEM_SEEN_QUOTA;
		ctx->quota_types |= QTYPE_MASK_GRP;
		break;
	case Opt_usrquota_block_hardlimit:
		size = memparse(param->string, &rest);
		if (*rest || !size)
			goto bad_value;
		if (size > SHMEM_QUOTA_MAX_SPC_LIMIT)
			return invalfc(fc,
				       "User quota block hardlimit too large.");
		ctx->qlimits.usrquota_bhardlimit = size;
		break;
	case Opt_grpquota_block_hardlimit:
		size = memparse(param->string, &rest);
		if (*rest || !size)
			goto bad_value;
		if (size > SHMEM_QUOTA_MAX_SPC_LIMIT)
			return invalfc(fc,
				       "Group quota block hardlimit too large.");
		ctx->qlimits.grpquota_bhardlimit = size;
		break;
	case Opt_usrquota_inode_hardlimit:
		size = memparse(param->string, &rest);
		if (*rest || !size)
			goto bad_value;
		if (size > SHMEM_QUOTA_MAX_INO_LIMIT)
			return invalfc(fc,
				       "User quota inode hardlimit too large.");
		ctx->qlimits.usrquota_ihardlimit = size;
		break;
	case Opt_grpquota_inode_hardlimit:
		size = memparse(param->string, &rest);
		if (*rest || !size)
			goto bad_value;
		if (size > SHMEM_QUOTA_MAX_INO_LIMIT)
			return invalfc(fc,
				       "Group quota inode hardlimit too large.");
		ctx->qlimits.grpquota_ihardlimit = size;
		break;
	case Opt_casefold_version:
		return shmem_parse_opt_casefold(fc, param, false);
	case Opt_casefold:
		return shmem_parse_opt_casefold(fc, param, true);
	case Opt_strict_encoding:
#if IS_ENABLED(CONFIG_UNICODE)
		ctx->strict_encoding = true;
		break;
#else
		return invalfc(fc, "tmpfs: Kernel not built with CONFIG_UNICODE\n");
#endif
	}
	return 0;

unsupported_parameter:
	return invalfc(fc, "Unsupported parameter '%s'", param->key);
bad_value:
	return invalfc(fc, "Bad value for '%s'", param->key);
}

static char *shmem_next_opt(char **s)
{
	char *sbegin = *s;
	char *p;

	if (sbegin == NULL)
		return NULL;

	/*
	 * NUL-terminate this option: unfortunately,
	 * mount options form a comma-separated list,
	 * but mpol's nodelist may also contain commas.
	 */
	for (;;) {
		p = strchr(*s, ',');
		if (p == NULL)
			break;
		*s = p + 1;
		if (!isdigit(*(p+1))) {
			*p = '\0';
			return sbegin;
		}
	}

	*s = NULL;
	return sbegin;
}

static int shmem_parse_monolithic(struct fs_context *fc, void *data)
{
	return vfs_parse_monolithic_sep(fc, data, shmem_next_opt);
}

/*
 * Reconfigure a shmem filesystem.
 */
static int shmem_reconfigure(struct fs_context *fc)
{
	struct shmem_options *ctx = fc->fs_private;
	struct shmem_sb_info *sbinfo = SHMEM_SB(fc->root->d_sb);
	unsigned long used_isp;
	struct mempolicy *mpol = NULL;
	const char *err;

	raw_spin_lock(&sbinfo->stat_lock);
	used_isp = sbinfo->max_inodes * BOGO_INODE_SIZE - sbinfo->free_ispace;

	if ((ctx->seen & SHMEM_SEEN_BLOCKS) && ctx->blocks) {
		if (!sbinfo->max_blocks) {
			err = "Cannot retroactively limit size";
			goto out;
		}
		if (percpu_counter_compare(&sbinfo->used_blocks,
					   ctx->blocks) > 0) {
			err = "Too small a size for current use";
			goto out;
		}
	}
	if ((ctx->seen & SHMEM_SEEN_INODES) && ctx->inodes) {
		if (!sbinfo->max_inodes) {
			err = "Cannot retroactively limit inodes";
			goto out;
		}
		if (ctx->inodes * BOGO_INODE_SIZE < used_isp) {
			err = "Too few inodes for current use";
			goto out;
		}
	}

	if ((ctx->seen & SHMEM_SEEN_INUMS) && !ctx->full_inums &&
	    sbinfo->next_ino > UINT_MAX) {
		err = "Current inum too high to switch to 32-bit inums";
		goto out;
	}

	/*
	 * "noswap" doesn't use fsparam_flag_no, i.e. there's no "swap"
	 * counterpart for (re-)enabling swap.
	 */
	if (ctx->noswap && !sbinfo->noswap) {
		err = "Cannot disable swap on remount";
		goto out;
	}

	if (ctx->seen & SHMEM_SEEN_QUOTA &&
	    !sb_any_quota_loaded(fc->root->d_sb)) {
		err = "Cannot enable quota on remount";
		goto out;
	}

#ifdef CONFIG_TMPFS_QUOTA
#define CHANGED_LIMIT(name)						\
	(ctx->qlimits.name## hardlimit &&				\
	(ctx->qlimits.name## hardlimit != sbinfo->qlimits.name## hardlimit))

	if (CHANGED_LIMIT(usrquota_b) || CHANGED_LIMIT(usrquota_i) ||
	    CHANGED_LIMIT(grpquota_b) || CHANGED_LIMIT(grpquota_i)) {
		err = "Cannot change global quota limit on remount";
		goto out;
	}
#endif /* CONFIG_TMPFS_QUOTA */

	if (ctx->seen & SHMEM_SEEN_HUGE)
		sbinfo->huge = ctx->huge;
	if (ctx->seen & SHMEM_SEEN_INUMS)
		sbinfo->full_inums = ctx->full_inums;
	if (ctx->seen & SHMEM_SEEN_BLOCKS)
		sbinfo->max_blocks  = ctx->blocks;
	if (ctx->seen & SHMEM_SEEN_INODES) {
		sbinfo->max_inodes  = ctx->inodes;
		sbinfo->free_ispace = ctx->inodes * BOGO_INODE_SIZE - used_isp;
	}

	/*
	 * Preserve previous mempolicy unless mpol remount option was specified.
	 */
	if (ctx->mpol) {
		mpol = sbinfo->mpol;
		sbinfo->mpol = ctx->mpol;	/* transfers initial ref */
		ctx->mpol = NULL;
	}

	if (ctx->noswap)
		sbinfo->noswap = true;

	raw_spin_unlock(&sbinfo->stat_lock);
	mpol_put(mpol);
	return 0;
out:
	raw_spin_unlock(&sbinfo->stat_lock);
	return invalfc(fc, "%s", err);
}

static int shmem_show_options(struct seq_file *seq, struct dentry *root)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(root->d_sb);
	struct mempolicy *mpol;

	if (sbinfo->max_blocks != shmem_default_max_blocks())
		seq_printf(seq, ",size=%luk", K(sbinfo->max_blocks));
	if (sbinfo->max_inodes != shmem_default_max_inodes())
		seq_printf(seq, ",nr_inodes=%lu", sbinfo->max_inodes);
	if (sbinfo->mode != (0777 | S_ISVTX))
		seq_printf(seq, ",mode=%03ho", sbinfo->mode);
	if (!uid_eq(sbinfo->uid, GLOBAL_ROOT_UID))
		seq_printf(seq, ",uid=%u",
				from_kuid_munged(&init_user_ns, sbinfo->uid));
	if (!gid_eq(sbinfo->gid, GLOBAL_ROOT_GID))
		seq_printf(seq, ",gid=%u",
				from_kgid_munged(&init_user_ns, sbinfo->gid));

	/*
	 * Showing inode{64,32} might be useful even if it's the system default,
	 * since then people don't have to resort to checking both here and
	 * /proc/config.gz to confirm 64-bit inums were successfully applied
	 * (which may not even exist if IKCONFIG_PROC isn't enabled).
	 *
	 * We hide it when inode64 isn't the default and we are using 32-bit
	 * inodes, since that probably just means the feature isn't even under
	 * consideration.
	 *
	 * As such:
	 *
	 *                     +-----------------+-----------------+
	 *                     | TMPFS_INODE64=y | TMPFS_INODE64=n |
	 *  +------------------+-----------------+-----------------+
	 *  | full_inums=true  | show            | show            |
	 *  | full_inums=false | show            | hide            |
	 *  +------------------+-----------------+-----------------+
	 *
	 */
	if (IS_ENABLED(CONFIG_TMPFS_INODE64) || sbinfo->full_inums)
		seq_printf(seq, ",inode%d", (sbinfo->full_inums ? 64 : 32));
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	/* Rightly or wrongly, show huge mount option unmasked by shmem_huge */
	if (sbinfo->huge)
		seq_printf(seq, ",huge=%s", shmem_format_huge(sbinfo->huge));
#endif
	mpol = shmem_get_sbmpol(sbinfo);
	shmem_show_mpol(seq, mpol);
	mpol_put(mpol);
	if (sbinfo->noswap)
		seq_printf(seq, ",noswap");
#ifdef CONFIG_TMPFS_QUOTA
	if (sb_has_quota_active(root->d_sb, USRQUOTA))
		seq_printf(seq, ",usrquota");
	if (sb_has_quota_active(root->d_sb, GRPQUOTA))
		seq_printf(seq, ",grpquota");
	if (sbinfo->qlimits.usrquota_bhardlimit)
		seq_printf(seq, ",usrquota_block_hardlimit=%lld",
			   sbinfo->qlimits.usrquota_bhardlimit);
	if (sbinfo->qlimits.grpquota_bhardlimit)
		seq_printf(seq, ",grpquota_block_hardlimit=%lld",
			   sbinfo->qlimits.grpquota_bhardlimit);
	if (sbinfo->qlimits.usrquota_ihardlimit)
		seq_printf(seq, ",usrquota_inode_hardlimit=%lld",
			   sbinfo->qlimits.usrquota_ihardlimit);
	if (sbinfo->qlimits.grpquota_ihardlimit)
		seq_printf(seq, ",grpquota_inode_hardlimit=%lld",
			   sbinfo->qlimits.grpquota_ihardlimit);
#endif
	return 0;
}

#endif /* CONFIG_TMPFS */

static void shmem_put_super(struct super_block *sb)
{
	struct shmem_sb_info *sbinfo = SHMEM_SB(sb);

#if IS_ENABLED(CONFIG_UNICODE)
	if (sb->s_encoding)
		utf8_unload(sb->s_encoding);
#endif

#ifdef CONFIG_TMPFS_QUOTA
	shmem_disable_quotas(sb);
#endif
	free_percpu(sbinfo->ino_batch);
	percpu_counter_destroy(&sbinfo->used_blocks);
	mpol_put(sbinfo->mpol);
	kfree(sbinfo);
	sb->s_fs_info = NULL;
}

#if IS_ENABLED(CONFIG_UNICODE) && defined(CONFIG_TMPFS)
static const struct dentry_operations shmem_ci_dentry_ops = {
	.d_hash = generic_ci_d_hash,
	.d_compare = generic_ci_d_compare,
};
#endif

static int shmem_fill_super(struct super_block *sb, struct fs_context *fc)
{
	struct shmem_options *ctx = fc->fs_private;
	struct inode *inode;
	struct shmem_sb_info *sbinfo;
	int error = -ENOMEM;

	/* Round up to L1_CACHE_BYTES to resist false sharing */
	sbinfo = kzalloc(max((int)sizeof(struct shmem_sb_info),
				L1_CACHE_BYTES), GFP_KERNEL);
	if (!sbinfo)
		return error;

	sb->s_fs_info = sbinfo;

#ifdef CONFIG_TMPFS
	/*
	 * Per default we only allow half of the physical ram per
	 * tmpfs instance, limiting inodes to one per page of lowmem;
	 * but the internal instance is left unlimited.
	 */
	if (!(sb->s_flags & SB_KERNMOUNT)) {
		if (!(ctx->seen & SHMEM_SEEN_BLOCKS))
			ctx->blocks = shmem_default_max_blocks();
		if (!(ctx->seen & SHMEM_SEEN_INODES))
			ctx->inodes = shmem_default_max_inodes();
		if (!(ctx->seen & SHMEM_SEEN_INUMS))
			ctx->full_inums = IS_ENABLED(CONFIG_TMPFS_INODE64);
		sbinfo->noswap = ctx->noswap;
	} else {
		sb->s_flags |= SB_NOUSER;
	}
	sb->s_export_op = &shmem_export_ops;
	sb->s_flags |= SB_NOSEC;

#if IS_ENABLED(CONFIG_UNICODE)
	if (!ctx->encoding && ctx->strict_encoding) {
		pr_err("tmpfs: strict_encoding option without encoding is forbidden\n");
		error = -EINVAL;
		goto failed;
	}

	if (ctx->encoding) {
		sb->s_encoding = ctx->encoding;
		set_default_d_op(sb, &shmem_ci_dentry_ops);
		if (ctx->strict_encoding)
			sb->s_encoding_flags = SB_ENC_STRICT_MODE_FL;
	}
#endif

#else
	sb->s_flags |= SB_NOUSER;
#endif /* CONFIG_TMPFS */
	sb->s_d_flags |= DCACHE_DONTCACHE;
	sbinfo->max_blocks = ctx->blocks;
	sbinfo->max_inodes = ctx->inodes;
	sbinfo->free_ispace = sbinfo->max_inodes * BOGO_INODE_SIZE;
	if (sb->s_flags & SB_KERNMOUNT) {
		sbinfo->ino_batch = alloc_percpu(ino_t);
		if (!sbinfo->ino_batch)
			goto failed;
	}
	sbinfo->uid = ctx->uid;
	sbinfo->gid = ctx->gid;
	sbinfo->full_inums = ctx->full_inums;
	sbinfo->mode = ctx->mode;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (ctx->seen & SHMEM_SEEN_HUGE)
		sbinfo->huge = ctx->huge;
	else
		sbinfo->huge = tmpfs_huge;
#endif
	sbinfo->mpol = ctx->mpol;
	ctx->mpol = NULL;

	raw_spin_lock_init(&sbinfo->stat_lock);
	if (percpu_counter_init(&sbinfo->used_blocks, 0, GFP_KERNEL))
		goto failed;
	spin_lock_init(&sbinfo->shrinklist_lock);
	INIT_LIST_HEAD(&sbinfo->shrinklist);

	sb->s_maxbytes = MAX_LFS_FILESIZE;
	sb->s_blocksize = PAGE_SIZE;
	sb->s_blocksize_bits = PAGE_SHIFT;
	sb->s_magic = TMPFS_MAGIC;
	sb->s_op = &shmem_ops;
	sb->s_time_gran = 1;
#ifdef CONFIG_TMPFS_XATTR
	sb->s_xattr = shmem_xattr_handlers;
#endif
#ifdef CONFIG_TMPFS_POSIX_ACL
	sb->s_flags |= SB_POSIXACL;
#endif
	uuid_t uuid;
	uuid_gen(&uuid);
	super_set_uuid(sb, uuid.b, sizeof(uuid));

#ifdef CONFIG_TMPFS_QUOTA
	if (ctx->seen & SHMEM_SEEN_QUOTA) {
		sb->dq_op = &shmem_quota_operations;
		sb->s_qcop = &dquot_quotactl_sysfile_ops;
		sb->s_quota_types = QTYPE_MASK_USR | QTYPE_MASK_GRP;

		/* Copy the default limits from ctx into sbinfo */
		memcpy(&sbinfo->qlimits, &ctx->qlimits,
		       sizeof(struct shmem_quota_limits));

		if (shmem_enable_quotas(sb, ctx->quota_types))
			goto failed;
	}
#endif /* CONFIG_TMPFS_QUOTA */

	inode = shmem_get_inode(&nop_mnt_idmap, sb, NULL,
				S_IFDIR | sbinfo->mode, 0,
				mk_vma_flags(VMA_NORESERVE_BIT));
	if (IS_ERR(inode)) {
		error = PTR_ERR(inode);
		goto failed;
	}
	inode->i_uid = sbinfo->uid;
	inode->i_gid = sbinfo->gid;
	sb->s_root = d_make_root(inode);
	if (!sb->s_root)
		goto failed;
	return 0;

failed:
	shmem_put_super(sb);
	return error;
}

static int shmem_get_tree(struct fs_context *fc)
{
	return get_tree_nodev(fc, shmem_fill_super);
}

static void shmem_free_fc(struct fs_context *fc)
{
	struct shmem_options *ctx = fc->fs_private;

	if (ctx) {
		mpol_put(ctx->mpol);
		kfree(ctx);
	}
}

static const struct fs_context_operations shmem_fs_context_ops = {
	.free			= shmem_free_fc,
	.get_tree		= shmem_get_tree,
#ifdef CONFIG_TMPFS
	.parse_monolithic	= shmem_parse_monolithic,
	.parse_param		= shmem_parse_one,
	.reconfigure		= shmem_reconfigure,
#endif
};

static struct kmem_cache *shmem_inode_cachep __ro_after_init;

static struct inode *shmem_alloc_inode(struct super_block *sb)
{
	struct shmem_inode_info *info;
	info = alloc_inode_sb(sb, shmem_inode_cachep, GFP_KERNEL);
	if (!info)
		return NULL;
	return &info->vfs_inode;
}

static void shmem_free_in_core_inode(struct inode *inode)
{
	if (S_ISLNK(inode->i_mode))
		kfree(inode->i_link);
	kmem_cache_free(shmem_inode_cachep, SHMEM_I(inode));
}

static void shmem_destroy_inode(struct inode *inode)
{
	if (S_ISREG(inode->i_mode))
		mpol_free_shared_policy(&SHMEM_I(inode)->policy);
	if (S_ISDIR(inode->i_mode))
		simple_offset_destroy(shmem_get_offset_ctx(inode));
}

static void shmem_init_inode(void *foo)
{
	struct shmem_inode_info *info = foo;
	inode_init_once(&info->vfs_inode);
}

static void __init shmem_init_inodecache(void)
{
	shmem_inode_cachep = kmem_cache_create("shmem_inode_cache",
				sizeof(struct shmem_inode_info),
				0, SLAB_PANIC|SLAB_ACCOUNT, shmem_init_inode);
}

static void __init shmem_destroy_inodecache(void)
{
	kmem_cache_destroy(shmem_inode_cachep);
}

/* Keep the page in page cache instead of truncating it */
static int shmem_error_remove_folio(struct address_space *mapping,
				   struct folio *folio)
{
	return 0;
}

static const struct address_space_operations shmem_aops = {
	.dirty_folio	= noop_dirty_folio,
#ifdef CONFIG_TMPFS
	.write_begin	= shmem_write_begin,
	.write_end	= shmem_write_end,
#endif
#ifdef CONFIG_MIGRATION
	.migrate_folio	= migrate_folio,
#endif
	.error_remove_folio = shmem_error_remove_folio,
};

static const struct file_operations shmem_file_operations = {
	.mmap_prepare	= shmem_mmap_prepare,
	.open		= shmem_file_open,
	.get_unmapped_area = shmem_get_unmapped_area,
#ifdef CONFIG_TMPFS
	.llseek		= shmem_file_llseek,
	.read_iter	= shmem_file_read_iter,
	.write_iter	= shmem_file_write_iter,
	.fsync		= noop_fsync,
	.splice_read	= shmem_file_splice_read,
	.splice_write	= iter_file_splice_write,
	.fallocate	= shmem_fallocate,
	.setlease	= generic_setlease,
#endif
};

static const struct inode_operations shmem_inode_operations = {
	.getattr	= shmem_getattr,
	.setattr	= shmem_setattr,
#ifdef CONFIG_TMPFS_XATTR
	.listxattr	= shmem_listxattr,
	.set_acl	= simple_set_acl,
	.fileattr_get	= shmem_fileattr_get,
	.fileattr_set	= shmem_fileattr_set,
#endif
};

static const struct inode_operations shmem_dir_inode_operations = {
#ifdef CONFIG_TMPFS
	.getattr	= shmem_getattr,
	.create		= shmem_create,
	.lookup		= simple_lookup,
	.link		= shmem_link,
	.unlink		= shmem_unlink,
	.symlink	= shmem_symlink,
	.mkdir		= shmem_mkdir,
	.rmdir		= shmem_rmdir,
	.mknod		= shmem_mknod,
	.rename		= shmem_rename2,
	.tmpfile	= shmem_tmpfile,
	.get_offset_ctx	= shmem_get_offset_ctx,
#endif
#ifdef CONFIG_TMPFS_XATTR
	.listxattr	= shmem_listxattr,
	.fileattr_get	= shmem_fileattr_get,
	.fileattr_set	= shmem_fileattr_set,
#endif
#ifdef CONFIG_TMPFS_POSIX_ACL
	.setattr	= shmem_setattr,
	.set_acl	= simple_set_acl,
#endif
};

static const struct inode_operations shmem_special_inode_operations = {
	.getattr	= shmem_getattr,
#ifdef CONFIG_TMPFS_XATTR
	.listxattr	= shmem_listxattr,
#endif
#ifdef CONFIG_TMPFS_POSIX_ACL
	.setattr	= shmem_setattr,
	.set_acl	= simple_set_acl,
#endif
};

static const struct super_operations shmem_ops = {
	.alloc_inode	= shmem_alloc_inode,
	.free_inode	= shmem_free_in_core_inode,
	.destroy_inode	= shmem_destroy_inode,
#ifdef CONFIG_TMPFS
	.statfs		= shmem_statfs,
	.show_options	= shmem_show_options,
#endif
#ifdef CONFIG_TMPFS_QUOTA
	.get_dquots	= shmem_get_dquots,
#endif
	.evict_inode	= shmem_evict_inode,
	.drop_inode	= inode_just_drop,
	.put_super	= shmem_put_super,
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	.nr_cached_objects	= shmem_unused_huge_count,
	.free_cached_objects	= shmem_unused_huge_scan,
#endif
};

static const struct vm_operations_struct shmem_vm_ops = {
	.fault		= shmem_fault,
	.map_pages	= filemap_map_pages,
#ifdef CONFIG_NUMA
	.set_policy     = shmem_set_policy,
	.get_policy     = shmem_get_policy,
#endif
};

static const struct vm_operations_struct shmem_anon_vm_ops = {
	.fault		= shmem_fault,
	.map_pages	= filemap_map_pages,
#ifdef CONFIG_NUMA
	.set_policy     = shmem_set_policy,
	.get_policy     = shmem_get_policy,
#endif
};

int shmem_init_fs_context(struct fs_context *fc)
{
	struct shmem_options *ctx;

	ctx = kzalloc_obj(struct shmem_options, GFP_KERNEL);
	if (!ctx)
		return -ENOMEM;

	ctx->mode = 0777 | S_ISVTX;
	ctx->uid = current_fsuid();
	ctx->gid = current_fsgid();

#if IS_ENABLED(CONFIG_UNICODE)
	ctx->encoding = NULL;
#endif

	fc->fs_private = ctx;
	fc->ops = &shmem_fs_context_ops;
#ifdef CONFIG_TMPFS
	fc->sb_flags |= SB_I_VERSION;
#endif
	return 0;
}

static struct file_system_type shmem_fs_type = {
	.owner		= THIS_MODULE,
	.name		= "tmpfs",
	.init_fs_context = shmem_init_fs_context,
#ifdef CONFIG_TMPFS
	.parameters	= shmem_fs_parameters,
#endif
	.kill_sb	= kill_anon_super,
	.fs_flags	= FS_USERNS_MOUNT | FS_ALLOW_IDMAP | FS_MGTIME,
};

#if defined(CONFIG_SYSFS) && defined(CONFIG_TMPFS)

#define __INIT_KOBJ_ATTR(_name, _mode, _show, _store)			\
{									\
	.attr	= { .name = __stringify(_name), .mode = _mode },	\
	.show	= _show,						\
	.store	= _store,						\
}

#define TMPFS_ATTR_W(_name, _store)				\
	static struct kobj_attribute tmpfs_attr_##_name =	\
			__INIT_KOBJ_ATTR(_name, 0200, NULL, _store)

#define TMPFS_ATTR_RW(_name, _show, _store)			\
	static struct kobj_attribute tmpfs_attr_##_name =	\
			__INIT_KOBJ_ATTR(_name, 0644, _show, _store)

#define TMPFS_ATTR_RO(_name, _show)				\
	static struct kobj_attribute tmpfs_attr_##_name =	\
			__INIT_KOBJ_ATTR(_name, 0444, _show, NULL)

#if IS_ENABLED(CONFIG_UNICODE)
static ssize_t casefold_show(struct kobject *kobj, struct kobj_attribute *a,
			char *buf)
{
		return sysfs_emit(buf, "supported\n");
}
TMPFS_ATTR_RO(casefold, casefold_show);
#endif

static struct attribute *tmpfs_attributes[] = {
#if IS_ENABLED(CONFIG_UNICODE)
	&tmpfs_attr_casefold.attr,
#endif
	NULL
};

static const struct attribute_group tmpfs_attribute_group = {
	.attrs = tmpfs_attributes,
	.name = "features"
};

static struct kobject *tmpfs_kobj;

static int __init tmpfs_sysfs_init(void)
{
	int ret;

	tmpfs_kobj = kobject_create_and_add("tmpfs", fs_kobj);
	if (!tmpfs_kobj)
		return -ENOMEM;

	ret = sysfs_create_group(tmpfs_kobj, &tmpfs_attribute_group);
	if (ret)
		kobject_put(tmpfs_kobj);

	return ret;
}
#endif /* CONFIG_SYSFS && CONFIG_TMPFS */

void __init shmem_init(void)
{
	int error;

	shmem_init_inodecache();

#ifdef CONFIG_TMPFS_QUOTA
	register_quota_format(&shmem_quota_format);
#endif

	error = register_filesystem(&shmem_fs_type);
	if (error) {
		pr_err("Could not register tmpfs\n");
		goto out2;
	}

	shm_mnt = kern_mount(&shmem_fs_type);
	if (IS_ERR(shm_mnt)) {
		error = PTR_ERR(shm_mnt);
		pr_err("Could not kern_mount tmpfs\n");
		goto out1;
	}

#if defined(CONFIG_SYSFS) && defined(CONFIG_TMPFS)
	error = tmpfs_sysfs_init();
	if (error) {
		pr_err("Could not init tmpfs sysfs\n");
		goto out1;
	}
#endif

#ifdef CONFIG_TRANSPARENT_HUGEPAGE
	if (has_transparent_hugepage() && shmem_huge > SHMEM_HUGE_DENY)
		SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge;
	else
		shmem_huge = SHMEM_HUGE_NEVER; /* just in case it was patched */

	/*
	 * Default to setting PMD-sized THP to inherit the global setting and
	 * disable all other multi-size THPs.
	 */
	if (!shmem_orders_configured)
		huge_shmem_orders_inherit = BIT(HPAGE_PMD_ORDER);
#endif
	return;

out1:
	unregister_filesystem(&shmem_fs_type);
out2:
#ifdef CONFIG_TMPFS_QUOTA
	unregister_quota_format(&shmem_quota_format);
#endif
	shmem_destroy_inodecache();
	shm_mnt = ERR_PTR(error);
}

#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && defined(CONFIG_SYSFS)
static ssize_t shmem_enabled_show(struct kobject *kobj,
				  struct kobj_attribute *attr, char *buf)
{
	static const int values[] = {
		SHMEM_HUGE_ALWAYS,
		SHMEM_HUGE_WITHIN_SIZE,
		SHMEM_HUGE_ADVISE,
		SHMEM_HUGE_NEVER,
		SHMEM_HUGE_DENY,
		SHMEM_HUGE_FORCE,
	};
	int len = 0;
	int i;

	for (i = 0; i < ARRAY_SIZE(values); i++) {
		len += sysfs_emit_at(buf, len,
				shmem_huge == values[i] ? "%s[%s]" : "%s%s",
				i ? " " : "", shmem_format_huge(values[i]));
	}
	len += sysfs_emit_at(buf, len, "\n");

	return len;
}

static ssize_t shmem_enabled_store(struct kobject *kobj,
		struct kobj_attribute *attr, const char *buf, size_t count)
{
	char tmp[16];
	int huge, err;

	if (count + 1 > sizeof(tmp))
		return -EINVAL;
	memcpy(tmp, buf, count);
	tmp[count] = '\0';
	if (count && tmp[count - 1] == '\n')
		tmp[count - 1] = '\0';

	huge = shmem_parse_huge(tmp);
	if (huge == -EINVAL)
		return huge;

	shmem_huge = huge;
	if (shmem_huge > SHMEM_HUGE_DENY)
		SHMEM_SB(shm_mnt->mnt_sb)->huge = shmem_huge;

	err = start_stop_khugepaged();
	return err ? err : count;
}

struct kobj_attribute shmem_enabled_attr = __ATTR_RW(shmem_enabled);
static DEFINE_SPINLOCK(huge_shmem_orders_lock);

static ssize_t thpsize_shmem_enabled_show(struct kobject *kobj,
					  struct kobj_attribute *attr, char *buf)
{
	int order = to_thpsize(kobj)->order;
	const char *output;

	if (test_bit(order, &huge_shmem_orders_always))
		output = "[always] inherit within_size advise never";
	else if (test_bit(order, &huge_shmem_orders_inherit))
		output = "always [inherit] within_size advise never";
	else if (test_bit(order, &huge_shmem_orders_within_size))
		output = "always inherit [within_size] advise never";
	else if (test_bit(order, &huge_shmem_orders_madvise))
		output = "always inherit within_size [advise] never";
	else
		output = "always inherit within_size advise [never]";

	return sysfs_emit(buf, "%s\n", output);
}

static ssize_t thpsize_shmem_enabled_store(struct kobject *kobj,
					   struct kobj_attribute *attr,
					   const char *buf, size_t count)
{
	int order = to_thpsize(kobj)->order;
	ssize_t ret = count;

	if (sysfs_streq(buf, "always")) {
		spin_lock(&huge_shmem_orders_lock);
		clear_bit(order, &huge_shmem_orders_inherit);
		clear_bit(order, &huge_shmem_orders_madvise);
		clear_bit(order, &huge_shmem_orders_within_size);
		set_bit(order, &huge_shmem_orders_always);
		spin_unlock(&huge_shmem_orders_lock);
	} else if (sysfs_streq(buf, "inherit")) {
		/* Do not override huge allocation policy with non-PMD sized mTHP */
		if (shmem_huge == SHMEM_HUGE_FORCE &&
		    order != HPAGE_PMD_ORDER)
			return -EINVAL;

		spin_lock(&huge_shmem_orders_lock);
		clear_bit(order, &huge_shmem_orders_always);
		clear_bit(order, &huge_shmem_orders_madvise);
		clear_bit(order, &huge_shmem_orders_within_size);
		set_bit(order, &huge_shmem_orders_inherit);
		spin_unlock(&huge_shmem_orders_lock);
	} else if (sysfs_streq(buf, "within_size")) {
		spin_lock(&huge_shmem_orders_lock);
		clear_bit(order, &huge_shmem_orders_always);
		clear_bit(order, &huge_shmem_orders_inherit);
		clear_bit(order, &huge_shmem_orders_madvise);
		set_bit(order, &huge_shmem_orders_within_size);
		spin_unlock(&huge_shmem_orders_lock);
	} else if (sysfs_streq(buf, "advise")) {
		spin_lock(&huge_shmem_orders_lock);
		clear_bit(order, &huge_shmem_orders_always);
		clear_bit(order, &huge_shmem_orders_inherit);
		clear_bit(order, &huge_shmem_orders_within_size);
		set_bit(order, &huge_shmem_orders_madvise);
		spin_unlock(&huge_shmem_orders_lock);
	} else if (sysfs_streq(buf, "never")) {
		spin_lock(&huge_shmem_orders_lock);
		clear_bit(order, &huge_shmem_orders_always);
		clear_bit(order, &huge_shmem_orders_inherit);
		clear_bit(order, &huge_shmem_orders_within_size);
		clear_bit(order, &huge_shmem_orders_madvise);
		spin_unlock(&huge_shmem_orders_lock);
	} else {
		ret = -EINVAL;
	}

	if (ret > 0) {
		int err = start_stop_khugepaged();

		if (err)
			ret = err;
	}
	return ret;
}

struct kobj_attribute thpsize_shmem_enabled_attr =
	__ATTR(shmem_enabled, 0644, thpsize_shmem_enabled_show, thpsize_shmem_enabled_store);
#endif /* CONFIG_TRANSPARENT_HUGEPAGE && CONFIG_SYSFS */

#if defined(CONFIG_TRANSPARENT_HUGEPAGE)

static int __init setup_transparent_hugepage_shmem(char *str)
{
	int huge;

	huge = shmem_parse_huge(str);
	if (huge == -EINVAL) {
		pr_warn("transparent_hugepage_shmem= cannot parse, ignored\n");
		return huge;
	}

	shmem_huge = huge;
	return 1;
}
__setup("transparent_hugepage_shmem=", setup_transparent_hugepage_shmem);

static int __init setup_transparent_hugepage_tmpfs(char *str)
{
	int huge;

	huge = shmem_parse_huge(str);
	if (huge < 0) {
		pr_warn("transparent_hugepage_tmpfs= cannot parse, ignored\n");
		return huge;
	}

	tmpfs_huge = huge;
	return 1;
}
__setup("transparent_hugepage_tmpfs=", setup_transparent_hugepage_tmpfs);

static char str_dup[PAGE_SIZE] __initdata;
static int __init setup_thp_shmem(char *str)
{
	char *token, *range, *policy, *subtoken;
	unsigned long always, inherit, madvise, within_size;
	char *start_size, *end_size;
	int start, end, nr;
	char *p;

	if (!str || strlen(str) + 1 > PAGE_SIZE)
		goto err;
	strscpy(str_dup, str);

	always = huge_shmem_orders_always;
	inherit = huge_shmem_orders_inherit;
	madvise = huge_shmem_orders_madvise;
	within_size = huge_shmem_orders_within_size;
	p = str_dup;
	while ((token = strsep(&p, ";")) != NULL) {
		range = strsep(&token, ":");
		policy = token;

		if (!policy)
			goto err;

		while ((subtoken = strsep(&range, ",")) != NULL) {
			if (strchr(subtoken, '-')) {
				start_size = strsep(&subtoken, "-");
				end_size = subtoken;

				start = get_order_from_str(start_size,
							   THP_ORDERS_ALL_FILE_DEFAULT);
				end = get_order_from_str(end_size,
							 THP_ORDERS_ALL_FILE_DEFAULT);
			} else {
				start_size = end_size = subtoken;
				start = end = get_order_from_str(subtoken,
								 THP_ORDERS_ALL_FILE_DEFAULT);
			}

			if (start < 0) {
				pr_err("invalid size %s in thp_shmem boot parameter\n",
				       start_size);
				goto err;
			}

			if (end < 0) {
				pr_err("invalid size %s in thp_shmem boot parameter\n",
				       end_size);
				goto err;
			}

			if (start > end)
				goto err;

			nr = end - start + 1;
			if (!strcmp(policy, "always")) {
				bitmap_set(&always, start, nr);
				bitmap_clear(&inherit, start, nr);
				bitmap_clear(&madvise, start, nr);
				bitmap_clear(&within_size, start, nr);
			} else if (!strcmp(policy, "advise")) {
				bitmap_set(&madvise, start, nr);
				bitmap_clear(&inherit, start, nr);
				bitmap_clear(&always, start, nr);
				bitmap_clear(&within_size, start, nr);
			} else if (!strcmp(policy, "inherit")) {
				bitmap_set(&inherit, start, nr);
				bitmap_clear(&madvise, start, nr);
				bitmap_clear(&always, start, nr);
				bitmap_clear(&within_size, start, nr);
			} else if (!strcmp(policy, "within_size")) {
				bitmap_set(&within_size, start, nr);
				bitmap_clear(&inherit, start, nr);
				bitmap_clear(&madvise, start, nr);
				bitmap_clear(&always, start, nr);
			} else if (!strcmp(policy, "never")) {
				bitmap_clear(&inherit, start, nr);
				bitmap_clear(&madvise, start, nr);
				bitmap_clear(&always, start, nr);
				bitmap_clear(&within_size, start, nr);
			} else {
				pr_err("invalid policy %s in thp_shmem boot parameter\n", policy);
				goto err;
			}
		}
	}

	huge_shmem_orders_always = always;
	huge_shmem_orders_madvise = madvise;
	huge_shmem_orders_inherit = inherit;
	huge_shmem_orders_within_size = within_size;
	shmem_orders_configured = true;
	return 1;

err:
	pr_warn("thp_shmem=%s: error parsing string, ignoring setting\n", str);
	return 0;
}
__setup("thp_shmem=", setup_thp_shmem);

#endif /* CONFIG_TRANSPARENT_HUGEPAGE */

#else /* !CONFIG_SHMEM */

/*
 * tiny-shmem: simple shmemfs and tmpfs using ramfs code
 *
 * This is intended for small system where the benefits of the full
 * shmem code (swap-backed and resource-limited) are outweighed by
 * their complexity. On systems without swap this code should be
 * effectively equivalent, but much lighter weight.
 */

static struct file_system_type shmem_fs_type = {
	.name		= "tmpfs",
	.init_fs_context = ramfs_init_fs_context,
	.parameters	= ramfs_fs_parameters,
	.kill_sb	= ramfs_kill_sb,
	.fs_flags	= FS_USERNS_MOUNT,
};

void __init shmem_init(void)
{
	BUG_ON(register_filesystem(&shmem_fs_type) != 0);

	shm_mnt = kern_mount(&shmem_fs_type);
	BUG_ON(IS_ERR(shm_mnt));
}

int shmem_unuse(unsigned int type)
{
	return 0;
}

int shmem_lock(struct file *file, int lock, struct ucounts *ucounts)
{
	return 0;
}

void shmem_unlock_mapping(struct address_space *mapping)
{
}

#ifdef CONFIG_MMU
unsigned long shmem_get_unmapped_area(struct file *file,
				      unsigned long addr, unsigned long len,
				      unsigned long pgoff, unsigned long flags)
{
	return mm_get_unmapped_area(file, addr, len, pgoff, flags);
}
#endif

void shmem_truncate_range(struct inode *inode, loff_t lstart, uoff_t lend)
{
	truncate_inode_pages_range(inode->i_mapping, lstart, lend);
}
EXPORT_SYMBOL_GPL(shmem_truncate_range);

#define shmem_vm_ops				generic_file_vm_ops
#define shmem_anon_vm_ops			generic_file_vm_ops
#define shmem_file_operations			ramfs_file_operations

static inline int shmem_acct_size(unsigned long flags, loff_t size)
{
	return 0;
}

static inline void shmem_unacct_size(unsigned long flags, loff_t size)
{
}

static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap,
				struct super_block *sb, struct inode *dir,
				umode_t mode, dev_t dev, vma_flags_t flags)
{
	struct inode *inode = ramfs_get_inode(sb, dir, mode, dev);
	return inode ? inode : ERR_PTR(-ENOSPC);
}

#endif /* CONFIG_SHMEM */

/* common code */

static struct file *__shmem_file_setup(struct vfsmount *mnt, const char *name,
				       loff_t size, vma_flags_t flags,
				       unsigned int i_flags)
{
	const unsigned long shmem_flags =
		vma_flags_test(&flags, VMA_NORESERVE_BIT) ? SHMEM_F_NORESERVE : 0;
	struct inode *inode;
	struct file *res;

	if (IS_ERR(mnt))
		return ERR_CAST(mnt);

	if (size < 0 || size > MAX_LFS_FILESIZE)
		return ERR_PTR(-EINVAL);

	if (is_idmapped_mnt(mnt))
		return ERR_PTR(-EINVAL);

	if (shmem_acct_size(shmem_flags, size))
		return ERR_PTR(-ENOMEM);

	inode = shmem_get_inode(&nop_mnt_idmap, mnt->mnt_sb, NULL,
				S_IFREG | S_IRWXUGO, 0, flags);
	if (IS_ERR(inode)) {
		shmem_unacct_size(shmem_flags, size);
		return ERR_CAST(inode);
	}
	inode->i_flags |= i_flags;
	inode->i_size = size;
	clear_nlink(inode);	/* It is unlinked */
	res = ERR_PTR(ramfs_nommu_expand_for_mapping(inode, size));
	if (!IS_ERR(res))
		res = alloc_file_pseudo(inode, mnt, name, O_RDWR,
				&shmem_file_operations);
	if (IS_ERR(res))
		iput(inode);
	return res;
}

/**
 * shmem_kernel_file_setup - get an unlinked file living in tmpfs which must be
 * 	kernel internal.  There will be NO LSM permission checks against the
 * 	underlying inode.  So users of this interface must do LSM checks at a
 *	higher layer.  The users are the big_key and shm implementations.  LSM
 *	checks are provided at the key or shm level rather than the inode.
 * @name: name for dentry (to be seen in /proc/<pid>/maps)
 * @size: size to be set for the file
 * @flags: VMA_NORESERVE_BIT suppresses pre-accounting of the entire object size
 */
struct file *shmem_kernel_file_setup(const char *name, loff_t size,
				     vma_flags_t flags)
{
	return __shmem_file_setup(shm_mnt, name, size, flags, S_PRIVATE);
}
EXPORT_SYMBOL_GPL(shmem_kernel_file_setup);

/**
 * shmem_file_setup - get an unlinked file living in tmpfs
 * @name: name for dentry (to be seen in /proc/<pid>/maps)
 * @size: size to be set for the file
 * @flags: VMA_NORESERVE_BIT suppresses pre-accounting of the entire object size
 */
struct file *shmem_file_setup(const char *name, loff_t size, vma_flags_t flags)
{
	return __shmem_file_setup(shm_mnt, name, size, flags, 0);
}
EXPORT_SYMBOL_GPL(shmem_file_setup);

/**
 * shmem_file_setup_with_mnt - get an unlinked file living in tmpfs
 * @mnt: the tmpfs mount where the file will be created
 * @name: name for dentry (to be seen in /proc/<pid>/maps)
 * @size: size to be set for the file
 * @flags: VMA_NORESERVE_BIT suppresses pre-accounting of the entire object size
 */
struct file *shmem_file_setup_with_mnt(struct vfsmount *mnt, const char *name,
				       loff_t size, vma_flags_t flags)
{
	return __shmem_file_setup(mnt, name, size, flags, 0);
}
EXPORT_SYMBOL_GPL(shmem_file_setup_with_mnt);

static struct file *__shmem_zero_setup(unsigned long start, unsigned long end,
		vma_flags_t flags)
{
	loff_t size = end - start;

	/*
	 * Cloning a new file under mmap_lock leads to a lock ordering conflict
	 * between XFS directory reading and selinux: since this file is only
	 * accessible to the user through its mapping, use S_PRIVATE flag to
	 * bypass file security, in the same way as shmem_kernel_file_setup().
	 */
	return shmem_kernel_file_setup("dev/zero", size, flags);
}

/**
 * shmem_zero_setup - setup a shared anonymous mapping
 * @vma: the vma to be mmapped is prepared by do_mmap
 * Returns: 0 on success, or error
 */
int shmem_zero_setup(struct vm_area_struct *vma)
{
	struct file *file = __shmem_zero_setup(vma->vm_start, vma->vm_end, vma->flags);

	if (IS_ERR(file))
		return PTR_ERR(file);

	if (vma->vm_file)
		fput(vma->vm_file);
	vma->vm_file = file;
	vma->vm_ops = &shmem_anon_vm_ops;

	return 0;
}

/**
 * shmem_zero_setup_desc - same as shmem_zero_setup, but determined by VMA
 * descriptor for convenience.
 * @desc: Describes VMA
 * Returns: 0 on success, or error
 */
int shmem_zero_setup_desc(struct vm_area_desc *desc)
{
	struct file *file = __shmem_zero_setup(desc->start, desc->end, desc->vma_flags);

	if (IS_ERR(file))
		return PTR_ERR(file);

	desc->vm_file = file;
	desc->vm_ops = &shmem_anon_vm_ops;

	return 0;
}

/**
 * shmem_read_folio_gfp - read into page cache, using specified page allocation flags.
 * @mapping:	the folio's address_space
 * @index:	the folio index
 * @gfp:	the page allocator flags to use if allocating
 *
 * This behaves as a tmpfs "read_cache_page_gfp(mapping, index, gfp)",
 * with any new page allocations done using the specified allocation flags.
 * But read_cache_page_gfp() uses the ->read_folio() method: which does not
 * suit tmpfs, since it may have pages in swapcache, and needs to find those
 * for itself; although drivers/gpu/drm i915 and ttm rely upon this support.
 *
 * i915_gem_object_get_pages_gtt() mixes __GFP_NORETRY | __GFP_NOWARN in
 * with the mapping_gfp_mask(), to avoid OOMing the machine unnecessarily.
 */
struct folio *shmem_read_folio_gfp(struct address_space *mapping,
		pgoff_t index, gfp_t gfp)
{
#ifdef CONFIG_SHMEM
	struct inode *inode = mapping->host;
	struct folio *folio;
	int error;

	error = shmem_get_folio_gfp(inode, index, i_size_read(inode),
				    &folio, SGP_CACHE, gfp, NULL, NULL);
	if (error)
		return ERR_PTR(error);

	folio_unlock(folio);
	return folio;
#else
	/*
	 * The tiny !SHMEM case uses ramfs without swap
	 */
	return mapping_read_folio_gfp(mapping, index, gfp);
#endif
}
EXPORT_SYMBOL_GPL(shmem_read_folio_gfp);

struct page *shmem_read_mapping_page_gfp(struct address_space *mapping,
					 pgoff_t index, gfp_t gfp)
{
	struct folio *folio = shmem_read_folio_gfp(mapping, index, gfp);
	struct page *page;

	if (IS_ERR(folio))
		return &folio->page;

	page = folio_file_page(folio, index);
	if (PageHWPoison(page)) {
		folio_put(folio);
		return ERR_PTR(-EIO);
	}

	return page;
}
EXPORT_SYMBOL_GPL(shmem_read_mapping_page_gfp);
]

gemini-3-flash-preview

							input: 117052
							output: 19
							thoughts: 3511
						

**My Analysis of a Potential Data Race in the Shmem Filesystem**

Okay, I've got a KCSAN report indicating a data race in the `shmem` filesystem, specifically on the `i_ctime_nsec` field of an inode.  Let me walk through my thought process. Initially, I saw some interesting code in `shmem_init_fs_context` and noted the absence of an `FS_MGTIME` flag set there but its presence in `shmem_fs_type.fs_flags`. However, I then went to review `shmem_fill_super` and observed that the `s_time_gran` is set to 1. My first thoughts centered around the possible race on timestamps and how the lack of the `FS_MGTIME` flag pointed to a race on the non-multigrain timestamp code.  The report suggests the issue is a standard data race on timestamps: reading `sec` and `nsec` without synchronization. As my analysis shows, this is a standard data race, leading to potentially inconsistent `sec`/`nsec` pairs in the time fields and potentially incorrect file timestamps.

I'm checking to see if this is considered a "benign" issue in the kernel.  The general consensus among developers seems to lean towards this being benign since it's "just a timestamp" and the inconsistencies are transient. However, KCSAN is intentionally designed to find these kinds of issues.  Looking at `generic_fillattr`, I can see it doesn't use any locking before accessing `stat->ctime`, using `inode_get_ctime`. `inode_set_ctime_current` (when not multigrain) also doesn't employ locking for the fields, although it might be called under an `i_rwsem`.  If it *were* multigrain timestamps, the kernel would use atomic operations via casting.  But that isn't the case here.

There's a well-known issue with racing timestamps.  There's a relevant patch series by Jeff Layton to address multigrain timestamps, but this particular issue happens even *without* multigrain timestamps. I'm trying to determine if it is harmful. For most applications, probably not. But for things like `make`, it could potentially cause issues because a file's timestamp may jump around.  However, the jumps are small and transient. I've rechecked `inode_set_ctime_to_ts` and `inode_get_ctime` to ensure the timestamps are read and written correctly.  The compiler could reorder the reads and writes, but even without it the described interleaving can happen. So, what about other synchronization primitives?  `shmem_xattr_handler_set` holds `inode->i_rwsem`, but `shmem_getattr` doesn't.  This confirms the data race.

Is it benign? By the definition, a benign race is on "simple" variables that don't need mutual exclusion.  Timestamps are *similar*, as metadata, not critical to control flow.  However, two fields (`sec` and `nsec`) are being modified, making it slightly more than a single "int". But, still, it's just a timestamp. I need to ensure that `i_ctime_nsec` is not used in other ways. In multigrain inodes, it stores the `I_CTIME_QUERIED` flag, but that isn't being used. It gets masked out in `inode_get_ctime_nsec` anyway. Thus, if the race is just on the timestamp, it's almost certainly benign, meaning no crashes.

I'm checking if there are other side effects, since `inode_inc_iversion(inode)` is called after `inode_set_ctime_current(inode)`. But, `inode_inc_iversion` uses atomic operations. The race is purely on `i_ctime_sec` and `i_ctime_nsec`. I'm going to bet it's considered benign by kernel maintainers, but KCSAN caught it because it is technically a race. Many similar races have been marked benign or fixed using `READ_ONCE`/`WRITE_ONCE` to silence KCSAN.

Looking at the KCSAN report, the `nsec` value changed, and by a considerable amount (over 500ms). If `sec` changed as well, we *could* have an inconsistency. But the report only shows a change in `nsec`. I need to ensure that the file system behavior is correct under these conditions. Does an incorrect timestamp returned by `stat()` break anything? Usually not. I've to confirm: no memory corruption, no crash risk. So, the code fits the benign race description.

I've to ensure that the writes are atomic. If not, the consistency will be broken.

map[Benign:true Confident:true]

map[Benign:true Confident:true]

gemini-3-flash-preview

							input: 121622
							output: 681
							thoughts: 157
						

**Benign Data Race in VFS Timestamp Update**

Alright, after digging into this, I'm pretty confident we've got a benign data race situation on our hands. My analysis shows it's all centered around the `i_ctime_nsec` field of an inode during timestamp updates. We're seeing the classic VFS behavior where the timestamp components, the `sec` and `nsec` fields, aren't being treated as an atomic pair for reads and writes.

I understand that this means `stat()` might occasionally report a timestamp that's transiently inconsistent, maybe showing an old seconds value combined with a new nanoseconds value or vice versa. But and this is key, this isn't leading to any real harm. I haven't found any evidence of memory corruption, crashes, or anything else that would keep me up at night.

Furthermore, I've confirmed that on the x86_64 architecture we're using, the field accesses themselves – the loads and stores – are atomic, whether they're 4-byte or 8-byte operations. We're dealing with a writer that has the inode mutex locked and a reader who has no locks, which is pretty standard for kernel attribute retrieval. Seems like a textbook case of a race that's unpleasant, but not dangerous. We can flag this behavior, but no action is needed here.