| Age | Commit message (Collapse) | Author | Lines |
|---|
|
In io-uring mode these requests are always re-used
and not allocated - we need to re-initialize them.
In order to set flags to zero a struct holding the flags
had to be added.
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
|
|
The FUSE protocol uses struct fuse_write_out to convey the return value of
copy_file_range, which is restricted to uint32_t. But the COPY_FILE_RANGE
interface supports a 64-bit size copies.
Currently the number of bytes copied is silently truncated to 32-bit, which
is unfortunate at best.
Implement the COPY_FILE_RANGE_64 interface which is identical to the old
one, except the number of bytes copied is returned in a 64-bit value.
The library interface remains the same.
If the kernel does not support the new interface or the server is running
as a 32-bit process, limit the copy size to size to UINT_MAX - 4096.
Edit by Bernd:
Keep ioctl_64bit and add use new bit is_copy_file_range_64 to keep
flags separated from each other - easier code readability IMO.
Reported-by: Florian Weimer <fweimer@redhat.com>
Closes: https://lore.kernel.org/all/lhuh5ynl8z5.fsf@oldenburg.str.redhat.com/
Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This is a long standing issue, a system could have unmounted
/path/to/mnt and then fuse-client/kernel would send FUSE_DESTROY,
which would then again try a umount.
Given that FUSE_DESTROY is async, that umount might arrive
any time later and might possibly unmount a wrong mount point.
A warning as in issue #1286 is just minor to that.
Code wise this uses atomics to free the char *, as FUSE_DESTROY
might race with a signal and a double free might be possible
without proctection. A lock might run into the same issue,
if the signal would arrive at the wrong time a double lock
would be possible.
Additionally, fuse_session_mount() is updated, to first
duplicatate the pointer and to then do the kernel mount -
reverting the kernel mount in case of strdup() failure
is much harder.
Closes: https://github.com/libfuse/libfuse/issues/1286
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
|
|
Signed-off-by: izxl007 <zeng.zheng@zte.com.cn>
|
|
fuse_apply_conn_info_opts() was applying to 'want_ext',
which would cause conflicts with 'want' if the application
applied its own flags to 'conn->want'.
Solution is:
- to move fuse_{set,unset,get}_feature_flag and
convert_to_conn_want_ext() to fuse_lowlevel.c and
to define them as part of the public API, although
convert_to_conn_want_ext() should not be used - it is
currently needed to be a public function due as it needs
to be defined for the tests.
Related to https://github.com/libfuse/libfuse/issues/1171 and
https://github.com/libfuse/libfuse/pull/1172.
Closes: https://github.com/libfuse/libfuse/issues/1171
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
there are too many issues with conn->want and conn->want_ext
conversion, for now just log a warning, but setting both
flags is now not fatal anymore.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Header/payload separation is part of the fuse-io-uring protocol
and might be later on for /dev/fuse legacy communication as well.
This is a preparation commit, for now fuse_ll_ops2 is unused.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Issue with previous code was that fuse_session_exit()
didn't wake up the semaphore in fuse_loop_mt.c.
Lock, semaphore and all uses of checking for "exited"
are now moved to struct fuse_session to have it
available for the signal handler.
This also removes internal fuse_session_reset() calls,
as that makes testing hard. From git history I also
don't see why it was added.
Closes: https://github.com/libfuse/libfuse/issues/997
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
The high-level interface triggers flag conversion twice: once in the
high-level init and once in the low-level init. This caused false
"both 'want' and 'want_ext' are set" errors when using
fuse_set_feature_flag() or fuse_unset_feature_flag().
The existing check for duplicate conversion only worked when 32-bit
flags were set directly. When using the preferred flag manipulation
functions, conn->want and the lower 32 bits of conn->want_ext
would differ, triggering the error.
Fix this by synchronizing conn->want with the lower 32 bits of
conn->want_ext after conversion, ensuring consistent state for
subsequent calls.
Closes: https://github.com/libfuse/libfuse/issues/1171
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
32-bit conn->want flags been left to be ABI compatible to 3.10,
even though the so version was changed.
The more recent way is to use fuse_set_feature_flag(),
which will use conn->want_ext.
Given that we now have two flags (want and want_ext), we need
to convert and that brought several issues
- If the application sets conn->want, that needs to be set into
the lower 32 bit of conn->want_ext. As the application might
actually unset values, it really has to be a copy and not
just 'or' - fixed now.
- convert_to_conn_want_ext() actually needs to check for
_modified_ conn->want and conn->want_ext
- convert_to_conn_want_ext() must consider being called from
high and lowlevel interfact, with different want_ext_default
and want_default values. It is only a failure, if the application
changed both, conn->want and conn->want_ext. This function
was failing in issue #1171, because high level fuse_fs_init()
was changing values and then lowlevel do_init() was incorrectly
failing on that.
This also adds a new test (test_want_conversion) and sets
values into example/{hello.c,hello_ll.c}
Also some more internal users of conn->want are converted to
fuse_{set,unset}_feature_flag().
Closes: https://github.com/libfuse/libfuse/issues/1171
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
|
|
se->buf_reallocable is true when reading /dev/fuse is handled
from internal functions - we can set the variable in
fuse_session_receive_buf_internal(). With that we also
don't need to have it an _Atomic variable anymore.
In _fuse_session_receive_buf() we can use "bool internal"
to check if the buffer can be re-allocated.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This fixes dynamic buffer allocation in commit
0e0f43b79b9b ("Reallocate fuse_session buffer...")
I noticed that when I increased the default fuse buf size as
possible in recent kernels.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
_int can be confused with 'integer'
|
|
Read/writes IOs should be page aligned as fuse server
might need to copy data to another buffer otherwise in
order to fulfill network or device storage requirements.
Simple reproducer is example/passthrough*
and opening a file with O_DIRECT - without this change
writing to that file failed with -EINVAL if the underlying
file system was using ext4 (for passthrough_hp the
'passthrough' feature has to be disabled).
The mis-alignment from fuse kernel is not ideal, but we can handle
it by allocation one page more than needed and then using a buffer
that is set up to compensate for kernel misalignment.
This also only set se->buf_reallocable to true when called
by a libfuse internal caller - we do not know what
external callers are doing with the buffer - update to
commit 0e0f43b79b9b
|
|
A previous PR supported extended max writes (eg write requests larger than 1 MB)
by initializing the fuse session buffer size to use the max_pages_limit set in
/proc/sys/fs/fuse. However, this is a huge problem for machines where multiple
fuse servers may be running but only one server needs large writes. In this case,
a lot of memory will be wasted and will lead to OOM issues.
This PR does a reallocation of the session buffer transparently if the server set
"se->conn.max_write" to a value larger than 1 MiB. This is only for buffers that
are "owned" by libfuse - if the server wishes to provide its own allocated buffer
for receiving/processing requests, then it should ensure that buffer is allocated
to the proper size from the start.
Local testing showed:
echo 65535 | sudo tee /proc/sys/fs/fuse/max_pages_limit
dd if=/dev/urandom of=hello_file bs=6M count=2
write requests:
write request size is 5242880
write request size is 1048576
write request size is 5242880
write request size is 1048576
|
|
max_write can be limited by se->op.init() and by the buffer size,
we use the minimum of these two.
Required se->bufsize is then set according to the determined
max_write. The current thread will have the old buffer size,
though, as it already had to the allocation to handle the
FUSE_INIT call (unless splice is used and ths variable
and related buffer is not used at all).
The given bufsize is just a hint for minimum size, allocation
could be actually larger (for example to get huge pages).
|
|
A recent upstream patch [1] changed FUSE_MAX_MAX_PAGES to
FUSE_DEFAULT_MAX_PAGES_LIMIT.
Update libfuse to use FUSE_DEFAULT_MAX_PAGES_LIMIT as well
instead of FUSE_MAX_MAX_PAGES.
[1] https://lore.kernel.org/linux-fsdevel/20240923171311.1561917-1-joannelkoong@gmail.com/T/#t
|
|
ref_cnt should make the intention of this variable more clear.
|
|
The variable is not modified exclusively with locks since commit
cef8c8b24902 ("Add support for no_interrupt") anymore.
That commit is safe, but might be error prone to future updates.
Changing it to a C11 _Atomic should avoid issues.
|
|
The API stays the same, the libfuse version comes from
inlined functions, which are defined fuse_lowlevel.h
and fuse.h. As these inlined functions are defined in the header
files they get added into the application, similar as if these
were preprocessor macros.
Macro vs inlined function is then just a style issue - I personally
prefer the latter.
fuse_session_new() -> static inlinei, in the application
_fuse_session_new -> inside of libfuse
fuse_new() -> static inline, in the application
_fuse_new() -> inside of libfuse
Note: Entirely untested is the fuse 30 api - we need a test
for it. And we do not have any ABI tests at all.
Signed-off-by: Bernd Schubert <bernd.schubert@fastmail.fm>
|
|
The io for FUSE requests and responses can now be further customized by allowing to write custom functions for reading/writing the responses. This includes overriding the splice io.
The reason for this addition is that having a custom file descriptor is not sufficient to allow custom io. Different types of file descriptor require different mechanisms of io interaction. For example, some file descriptor communication has boundaries (SOCK_DGRAM, EOF, etc...), while other types of fd:s might be unbounded (SOCK_STREAMS, ...). For unbounded communication, you have to read the header of the FUSE request first, and then read the remaining packet data. Furthermore, the one read call does not necessarily return all the data expected, requiring further
calls in a loop.
|
|
On benchmarking metadata operations with a single threaded bonnie++
and "max_idle_threads" limited to 1, 'top' was showing suspicious
160% cpu usage.
Profiling the system with flame graphs showed that an astonishing
amount of CPU time was spent in thread creation and destruction.
After verifying the code it turned out that fuse_do_work() was
creating a new thread every time all existing idle threads
were already busy. And then just a few lines later after processing
the current request it noticed that it had created too many threads
and destructed the current thread. I.e. there was a thread
creation/destruction ping-pong.
Code is changed to only create new threads if the max number of
threads is not reached.
Furthermore, thread destruction is disabled, as creation/destruction
is expensive in general.
With this change cpu usage of passthrough_hp went from ~160% to
~80% (with different values of max_idle_threads). And bonnie
values got approximately faster by 90%. This is a with single
threaded bonnie++
bonnie++ -x 4 -q -s0 -d <path> -n 30:1:1:10 -r 0
Without this patch, using the default max_idle_threads=10 and just
a single bonnie++ the thread creation/destruction code path is not
triggered. Just one libfuse and one application thread is just
a corner case - the requirement for the issue was just
n-application-threads >= max_idle_threads.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
struct fuse_loop_config was passed as a plain struct, without any
version identifer. This had two implications
1) Any addition of new parameters required a FUSE_SYMVER for
fuse_session_loop_mt() and fuse_loop_mt() as otherwise a read
beyond end-of previous struct size might have happened.
2) Filesystems also might have been recompiled and the developer
might not have noticed the struct extensions and unexpected for
the developer (or people recomliling the code) uninitialized
parameters would have been passed.
Code is updated to have struct fuse_loop_config as an opaque/private
data type for file systems that want version 312
(FUSE_MAKE_VERSION(3, 12)). The deprecated fuse_loop_config_v1
is visible, but should not be used outside of internal
conversion functions
File systems that want version >= 32 < 312 get the previous
struct (through ifdefs) and the #define of fuse_loop_mt
and fuse_session_loop_mt ensures that these recompiled file
systems call into the previous API, which then converts
the struct. This is similar to existing compiled applications
when just libfuse updated, but binaries it is solved with
the FUSE_SYMVER ABI compact declarations.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Applications may wish to call fuse_log() for unified logging. This way
they don't need to define their own wrappers to invoke the log message
handler function installed by fuse_set_log_func().
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
|
|
Introduce an API for custom log handler functions. This allows libfuse
applications to send messages to syslog(3) or other logging systems.
See include/fuse_log.h for details.
Convert libfuse from fprintf(stderr, ...) to log_fuse(level, ...). Most
messages are error messages with FUSE_LOG_ERR log level. There are also
some debug messages which now use the FUSE_LOG_DEBUG log level.
Note that lib/mount_util.c is used by both libfuse and fusermount3.
Since fusermount3 does not link against libfuse, we cannot call
fuse_log() from lib/mount_util.c. This file will continue to use
fprintf(stderr, ...) until someone figures out how to split it up.
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
|
|
Starting with kernel version 4.20 fuse supports a new property
'max_pages' which is the maximum number of pages that can be used per
request. This can be set via an argument during initialization.
This new property allows writes to be larger than 128k.
This patch sets the property if the matching capability is set
(FUSE_MAX_PAGES). It will also set max_write to 1MiB. Filesystems have
the possibility to decrease this size by setting max_write to a smaller
size. The max_pages and bufsize fields are adjusted accordingly.
Cc: Constantine Shulyupin <const@MakeLinux.com>
Signed-off-by: Markus Pargmann <scosu@quobyte.com>
|
|
The fuse_session_loop_mt() and fuse_loop_mt() symbols are only visible
when linking against the shared object. The code in lib/, however, is
compiled *into* the shared object and should thus use the internal
names of these functions.
Surprisingly enough, the code still worked before - but only when link
time optimization was disabled.
Unfortunately, we still can't compile with LTO because it seems that
enabling LTO somehow makes the tagged symbols vanish.
Without lto, we have:
$ nm lib/libfuse3.so | grep fuse_new
0000000000011070 T fuse_new_30
0000000000010a00 t fuse_new_31
0000000000011070 T fuse_new@FUSE_3.0
0000000000010a00 T fuse_new@@FUSE_3.1
and with LTO:
$ nm lib/libfuse3.so | grep fuse_new
0000000000019a70 T fuse_new_30
0000000000019270 t fuse_new_31
See also issue #198.
|
|
this fixes building with lto, which failed since commit 503e32d01e4db00e90d7acfd81ab05386559069f
|
|
-oallow_root is handled in userspace, and requires passing -oallow_other
to the kernel. This patch should make the code easier to understand and
avoid the confusion that gave rise to issue #86.
|
|
|
|
Eventually, this setting should be negotiated in the filesystem's init()
handler (like e.g. max_write). However, this requires corresponding
changes in the FUSE kernel module. In preparation for this (and to allow
a transition period) we already allow (and require) filesystems to set
the value in the init() handler in addition to the mount option.
The end-goal is tracked in issue #91.
|
|
Both the BSD and Linux implementation actually accept mostly the same
FUSE-specific mount options. Up to now, the BSD help function appended
the output of ``mount_fusefs --help``, but looking at
http://www.unix.com/man-page/freebsd/8/mount_fusefs/ this is likely more
confusing than helpful (since the user is not actually invoking
mount_fusefs directly, most of the options don't make sense).
|
|
Instead of using command line options to modify struct fuse_conn_info
before and after calling the init() handler, we now give the file system
explicit control over this.
|
|
This option really affects the behavior of the session loop, not the
low-level interface. Therefore, it does not belong in the fuse_session
object.
|
|
The session options are used only once to determine the proper
conn->want flags. It is nice to have them clearly separated from the
other struct fuse_session members that are used throughout the life of
the file system.
|
|
This is redundant with the capability flags in `wants` and `capable`.
|
|
This option is obsolete and should always be enabled. File systems that
want to limit the size of write requests should use the
``-o max_write=<N>`` option instead.
|
|
|
|
Merged the structures, and replaced fuse_ll with fuse_session
in all type definitions.
|
|
This is a code simplification patch.
- It confines most of the implementation channel implementation into
fuse_loop_mt (which is its only user).
- It makes it more obvious in the code that channels are only ever used
when using -o clone_fd and multi-threaded main loop.
- It simplies the definition of both struct fuse_session and struct
fuse_chan.
- Theoretically it should result in (minuscule) performance
improvements when not using -o clone_fd.
- Overall, it removes a lot more lines of source code than it adds :-).
|
|
|
|
The only struct fuse_chan that's accessible to the user application is
the "master" channel that is returned by fuse_mount and stored in struct
fuse_session.
When using the multi-threaded main loop with the "clone_fd" option, each
worker thread gets its own struct fuse_chan. However, none of these are
available to the user application, nor do they hold references to struct
fuse_session (the pointer is always null).
Therefore, any presence of struct fuse_chan can be removed
without loss of functionality by relying on struct fuse_session instead.
This reduces the number of API functions and removes a potential source
of confusion (since the new API no longer looks as if it might be
possible to add multiple channels to one session, or to share one
channel between multiple sessions).
Fixes issue #17.
|
|
|
|
This function is only used in one place.
|
|
This creates a separate device file descriptor for each processing thread,
which might improve performance.
|
|
New functions: fuse_chan_get(), fuse_chan_put(). Removed function:
fuse_chan_destroy().
|
|
|
|
Asynchronous direct I/O is supported by linux kernels 3.13 and
later, writeback caching is supported by 3.14 and later.
|
