| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Commit f8fe398ee14864e2c3c7c524ca851d7cc08bed28 introduced an invalid
passthrough_hp.cc indentation and that was not detected by checkpatch,
as that only handles C files.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
With io-uring the req owns the payload buffer, the application
can directly access it and copy data into it.
fuse_buf_copy_one() already has a check for dstmem == srcmem
and skips data copies.
fuse_reply_data
fuse_reply_data_uring
fuse_buf_copy
fuse_buf_copy_one
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Move fallocate implementation to passthrough_helpers.h, so that
it could be reused by multiple passthrough examples.
Signed-off-by: CismonX <admin@cismon.net>
|
|
Signed-off-by: izxl007 <zeng.zheng@zte.com.cn>
|
|
Signed-off-by: Gleb Popov <6yearold@gmail.com>
|
|
Signed-off-by: Gleb Popov <6yearold@gmail.com>
|
|
Signed-off-by: Gleb Popov <6yearold@gmail.com>
|
|
Signed-off-by: Gleb Popov <6yearold@gmail.com>
|
|
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Not ideal for git history, but the different
formatting within the project is really
disturbing.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This patch updates how to use the io_uring and io_uring_q_depth mount
options. There are two reasons for this:
1. It is recommended for end-users to use a format like “-o io_uring"
(with a space) in README.fuse-io-uring. This is because, in the
passthrough_hp example, cxxopts does not support the "-oio_uring"
format (without a space), and using it would result in a syntax error.
2. Remove redundant "--uring" and "--uring-q-depth=" from
passthrough_hp example.
Signed-off-by: izxl007 <zeng.zheng@zte.com.cn>
|
|
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
These are several buf size fixes
1) Set bufsize when HAVE_SPLICE is not defined. Addresses
https://github.com/libfuse/libfuse/issues/1184
2) Check in the read retry condition for bufsize, i.e. the value
passed to read and not for the buf->mem_size. Using buf->mem_size
can be startup racy. Additionally we now also set bufsize on
allocation to avoid these races.
3) Allocation and value assigned needs to follow the pattern
bufsize = se->bufsize;
buf_alloc(bufsize, internal)
buf->mem_size = bufsize;
I.e. bufsize has to be retrieved first, as se->bufsize might
change at anytime - the value used for allocation and
must not differ from the value actually used.
This also tries to set large sizes in passthrough_hp, to catch
issues in xfstests - though requires to set
/proc/sys/fs/fuse/max_pages_limit
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This commit enables splice moves by default, unless the caller opts into
nosplice.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
|
|
Note that name hashes and using paths as parameters
makes it very hard to support
anonymous files in the high level API.
Known Issues:
- tests have to bail out when O_TMPFILE is not supported.
This will always be the case with high level passthrough implementations.
- test_create_and_link_tmpfile has to be skipped
due to unidentified problems with github runner
|
|
This should stop some more xfstest test failures.
|
|
Simplify setting feature flags a bit by adding a helper
function.
Also move the check for valid flags into a funtion we can re-use
in a later patch.
|
|
fuse_loop_cfg_set_idle_threads() was by accident and
setting it might cause a performance issue.
|
|
These flags should be also set for O_CREAT
|
|
sfs_open and sfs_create set fi->direct_io (FOPEN_DIRECT_IO) when
O_DIRECT is given, in order to benefit from a shared inode lock
in kernel, i.e. to get parallel DIO writes. However, kernel side
disabled passthrough when FOPEN_DIRECT_IO is set. Reads/writes
had been totally failing in this case for O_DIRECT as
sfs_write_buf() and sfs_read() have a sanity check. That sanity
check could be modified, but for performance passthrough is
better than parallel DIO, hence, we only want automatic
FOPEN_DIRECT_IO for O_DIRECT when passthrough is not enabled.
Fixes: https://github.com/libfuse/libfuse/issues/1027
This also fixes automatically switching to FOPEN_DIRECT_IO
for O_DIRECT in sfs_create().
|
|
High level examples were already using it, but not
lowlevel. Also update the documentation.
|
|
Fix some clang-tidy warnings.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Commit 170edc6a8ef0 added dot and dotdot (. and ..) to readdir
results, but introduced an issue when max number of entries
was reached - lookup count must not be decreased without
doing the lookup.
With ext4 as underlying file system readir seems to return . and ..
at random offsets and randomly failed xfstests for me.
This also fixes indentation, as passthrough_hp.cc does not follow
the linux indentation style (if we decide to fix this, it needs
to be done for the entire file).
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
I see random ENOTCONN failures in xfstest generic/013 and generic/014
in my branch, but earliest on the 2nd run - takes ~12hours to get
the issue, but then there are no further information logged.
ENOTCONN points to a daemon crash - I need backtraces and a core dump.
This adds optional handling of fatal signals to print a core dump
and optional syslog logging with these new public functions:
fuse_set_fail_signal_handlers()
In addition to the existing fuse_set_signal_handlers(). This is not
enabled together with fuse_set_signal_handlers(), as it is change
in behavior and file systems might already have their own fatal
handlers.
fuse_log_enable_syslog
Print logs to syslog instead of stderr
fuse_log_close_syslog
Close syslog (for now just does closelog())
Code in fuse_signals.c is also updated, to be an array of signals,
and setting signal handlers is now down with a for-loop instead
of one hand coded set_one_signal_handler() per signal.
|
|
generic/401 fails currently because it checks that "." and ".." are
listed as directory entries.
Include "." and ".." as listed directory entries in passthrough_hp's
readdir implementation.
Signed-off by: Joanne Koong <joannelkoong@gmail.com>
|
|
generic/003 fails currently because if you specify -o relatime we will
fail to mount, but not return an error, so the test thinks that we
support relatime and then hilarity ensues. Set ret so that if we get
any failures while trying to mount we will properly error out.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
|
|
The function fuse_session_process_buf_int() would do much things
for FUSE_INTERRUPT requests, even there are no FUSE_INTERRUPT requests:
1. check every non-FUSE_INTERRUPT request and add these requests to the
linked list(se->list) under a big lock(se->lock).
2. the function fuse_free_req() frees every request and remove them from
the linked list(se->list) under a bing lock(se->lock).
These operations are not meaningful when there are no FUSE_INTERRUPT requests,
and have a great impact on the performance of fuse filesystem because the big
lock for each request.
In some cases, FUSE_INTERRUPT requests are infrequent, even none at all.
Besides, the user-defined filesystem may do nothing for FUSE_INTERRUPT requests.
And the kernel side has the option "no_interrupt" in struct fuse_conn. This kernel option
can be enabled by return ENOSYS in libfuse for the reply of FUSE_INTERRUPT request.
But I don't find the code to enable the "no_interrupt" kernel option in libfuse.
So add the no_interrupt support, and when this operaion is enabled:
1. remove the useless locking operaions and list operations.
2. return ENOSYS for the reply of FUSE_INTERRUPT request to inform the kernel to disable
FUSE_INTERRUPT request.
|
|
Add support for filesystem passthrough read/write of files.
When the FUSE_PASSTHROUGH capability is enabled, the FUSE server may
decide, while handling the "open" or "create" requests, if the given
file can be accessed by that process in "passthrough" mode, meaning that
all the further read and write operations would be forwarded by the
kernel directly to the backing file rather than to the FUSE server.
All requests other than read or write are still handled by the server.
This allows for an improved performance on reads and writes, especially
in the case of reads at random offsets, for which no (readahead)
caching mechanism would help, reducing the performance gap between FUSE
and native filesystem access.
Extend also the passthrough_hp example with the new passthrough feature.
This example opens a kernel backing file per FUSE inode on the first
FUSE file open of that inode and closes the backing file on the release
of the last FUSE file on that inode.
All opens of the same inode passthrough to the same backing file.
A combination of fi->direct_io and fi->passthrough is allowed.
It means that read/write operations go directly to the server, but mmap
is done on the backing file.
This allows to open some fds of the inode in passthrough mode and some
fd of the same inode in direct_io/passthrough_mmap mode.
Signed-off-by: Alessio Balsini <balsini@android.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
Add more documentation for FUSE_CAP_EXPORT_SUPPORT
Also remove the flag from passthrough_ll.c and passthrough_hp.cc
as these implementations do _not_ handle that flag. They just
cast fuse_ino_t to an inode and cause a heap buffer overflow
for unknown objects (simplest reproducer are the examples
in "man 2 open_by_handle_at", but to unmount/mount the file
system after name_to_handle_at and before open_by_handle_at).
Fixes https://github.com/libfuse/libfuse/issues/838
---------
Co-authored-by: Nikolaus Rath <Nikolaus@rath.org>
|
|
Shared locks (parallel_direct_writes) cannot be enabled for O_DIRECT, as
O_DIRECT may be set past file open time with fcntl(fd, F_SETFD, ...).
Kernel side fuse has precautions for shared lock direct-IO (direct_io in
libfuse), as it needs an exclusive inode lock when direct and page cache
IO happend at the same time.
In order to enjoy the parallel_direct_writes feature (i.e., get a shared
lock, not exclusive lock) for writes to the same file), direct_io is needed.
The feature direct_io is corresponding to FOPEN_DIRECT_IO in fuse kernel.
FOPEN_DIRECT_IO and O_DIRECT are not entirely the same as described above.
So enable direct_io (i.e., FOPEN_DIRECT_IO in fuse kernel) to enjoy parallel
direct_writes.
Some patches related to FOPEN_DIRECT_IO and O_DIRECT are below:
https://lore.kernel.org/all/753d6823-e984-4730-a126-d66b65ea772c@ddn.com
|
|
Move the parallel_direct_writes enable action to the init function in high level API,
it is more recommended just like commit 8ee553dac. Besides, add some comments to show
that the feature parallel_direct_writes is depend on the feature direct_io (refer to
kernel side patch series to consolidate direct IO, link: https://lwn.net/ml/linux-fsdevel/
20230918150313.3845114-1-bschubert@ddn.com for the reason).
|
|
This is not called FUSE_CAP_DIRECT_IO_RELAX, as the kernel flag
FUSE_DIRECT_IO_RELAX is supposed to be renamed to
FUSE_DIRECT_IO_ALLOW_MMAP. The corresponding kernel patches just
did not land yet.
|
|
The clone-fd option is set valued but not used in the context. Use it in the code.
|
|
All these passthrough examples don't need writes to be serialized.
Actually, most file systems probably handle non serialized parallel
direct writes - the FOPEN_PARALLEL_DIRECT_WRITES flag is just
to avoid a regression for those file system that rely on serialized
DIO writes in fuse kernel. Passthrough file system forward the IO
to another file system, which actually handles that internally -
serialized in fuser kernel is not needed.
|
|
|
|
Actually one had to use --clone-fd=1 instead of
just --clone-fd.
|
|
this is needed to test FOPEN_DIRECT_IO with xfstests. Also useful
for some benchmarks.
|
|
|
|
fuse_session_mount() may print errors to stderr, if we daemonize before
that than these are lost.
|
|
|
|
This addresses: https://github.com/libfuse/libfuse/issues/724
HAVE_LIBC_VERSIONED_SYMBOLS configures the library if to use
versioned symbols and is set at meson configuration time.
External filesystems (the main target, actually)
include fuse headers and the preprocessor
then acts on HAVE_LIBC_VERSIONED_SYMBOLS. Problem was now that
'config.h' was not distributed with libfuse and so
HAVE_LIBC_VERSIONED_SYMBOLS was never defined with external
tools and the preprocessor did the wrong decision.
This commit also increases the the minimal meson version,
as this depends on meson feature only available in 0.50
<quote 'meson' >
WARNING: Project specifies a minimum meson_
version '>= 0.42' but uses features which were added
in newer versions:
* 0.50.0: {'install arg in configure_file'}
</quote>
Additionally the config file has been renamed to "fuse_config.h"
to avoid clashes - 'config.h' is not very specific.
|
|
Just a slight code simplification.
|
|
This is useful for benchmarking.
Note: This changes behavior - passthrough_hp runs in background by default
now.
|
|
On benchmarking metadata operations with a single threaded bonnie++
and "max_idle_threads" limited to 1, 'top' was showing suspicious
160% cpu usage.
Profiling the system with flame graphs showed that an astonishing
amount of CPU time was spent in thread creation and destruction.
After verifying the code it turned out that fuse_do_work() was
creating a new thread every time all existing idle threads
were already busy. And then just a few lines later after processing
the current request it noticed that it had created too many threads
and destructed the current thread. I.e. there was a thread
creation/destruction ping-pong.
Code is changed to only create new threads if the max number of
threads is not reached.
Furthermore, thread destruction is disabled, as creation/destruction
is expensive in general.
With this change cpu usage of passthrough_hp went from ~160% to
~80% (with different values of max_idle_threads). And bonnie
values got approximately faster by 90%. This is a with single
threaded bonnie++
bonnie++ -x 4 -q -s0 -d <path> -n 30:1:1:10 -r 0
Without this patch, using the default max_idle_threads=10 and just
a single bonnie++ the thread creation/destruction code path is not
triggered. Just one libfuse and one application thread is just
a corner case - the requirement for the issue was just
n-application-threads >= max_idle_threads.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
passthrough_hp was not updated when splice got enabled by default in libfuse3.
I.e. the --nosplice option and condition on it was a noop.
|
|
sfs_unlink may call do_lookup(), which increases the inode ref count,
but since that function does not return attributes that lookup ref
count won't get automatically decreased.
|
|
Allow requesting from kernel to avoid flush on close at file open
time. If kernel does not support FOPEN_NOFLUSH flag, the request
will be ignored.
For passthrough_hp example, request to avoid flush on close when
writeback cache is disabled and file is opened O_RDONLY.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
Before last unlink() release the reference on inode.fd to allow reuse
of underlying fs inode number, mark the server inode "deleted" and bump
it's generation counter.
When same inode number is found on lookup(), the server inode object will
be reused as well.
Skip this when inode has an open file and when writeback cache is enabled.
This will be used to verify inode reuse bug fix in the kernel.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
|
|
* passthrough_ll/hp: remove symlink fallbacks
Path lookup in the kernel has special rules for looking up magic symlinks
under /proc. If a filesystem operation is instructed to follow symlinks
(e.g. via AT_SYMLINK_FOLLOW or lack of AT_SYMLINK_NOFOLLOW), and the final
component is such a proc symlink, then the target of the magic symlink is
used for the operation, even if the target itself is a symlink. I.e. path
lookup is always terminated after following a final magic symlink.
I was erronously assuming that in the above case the target symlink would
also be followed, and so workarounds were added for a couple of operations
to handle the symlink case. Since the symlink can be handled simply by
following the proc symlink, these workardouds are not needed.
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Co-authored-by: Miklos Szeredi <mszeredi@redhat.com>
|
