| Age | Commit message (Collapse) | Author | Lines |
|---|
|
The high-level interface triggers flag conversion twice: once in the
high-level init and once in the low-level init. This caused false
"both 'want' and 'want_ext' are set" errors when using
fuse_set_feature_flag() or fuse_unset_feature_flag().
The existing check for duplicate conversion only worked when 32-bit
flags were set directly. When using the preferred flag manipulation
functions, conn->want and the lower 32 bits of conn->want_ext
would differ, triggering the error.
Fix this by synchronizing conn->want with the lower 32 bits of
conn->want_ext after conversion, ensuring consistent state for
subsequent calls.
Closes: https://github.com/libfuse/libfuse/issues/1171
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This commit enables splice moves by default, unless the caller opts into
nosplice.
Signed-off-by: Joanne Koong <joannelkoong@gmail.com>
|
|
These are several buf size fixes
1) Set bufsize when HAVE_SPLICE is not defined. Addresses
https://github.com/libfuse/libfuse/issues/1184
2) Check in the read retry condition for bufsize, i.e. the value
passed to read and not for the buf->mem_size. Using buf->mem_size
can be startup racy. Additionally we now also set bufsize on
allocation to avoid these races.
3) Allocation and value assigned needs to follow the pattern
bufsize = se->bufsize;
buf_alloc(bufsize, internal)
buf->mem_size = bufsize;
I.e. bufsize has to be retrieved first, as se->bufsize might
change at anytime - the value used for allocation and
must not differ from the value actually used.
This also tries to set large sizes in passthrough_hp, to catch
issues in xfstests - though requires to set
/proc/sys/fs/fuse/max_pages_limit
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
32-bit conn->want flags been left to be ABI compatible to 3.10,
even though the so version was changed.
The more recent way is to use fuse_set_feature_flag(),
which will use conn->want_ext.
Given that we now have two flags (want and want_ext), we need
to convert and that brought several issues
- If the application sets conn->want, that needs to be set into
the lower 32 bit of conn->want_ext. As the application might
actually unset values, it really has to be a copy and not
just 'or' - fixed now.
- convert_to_conn_want_ext() actually needs to check for
_modified_ conn->want and conn->want_ext
- convert_to_conn_want_ext() must consider being called from
high and lowlevel interfact, with different want_ext_default
and want_default values. It is only a failure, if the application
changed both, conn->want and conn->want_ext. This function
was failing in issue #1171, because high level fuse_fs_init()
was changing values and then lowlevel do_init() was incorrectly
failing on that.
This also adds a new test (test_want_conversion) and sets
values into example/{hello.c,hello_ll.c}
Also some more internal users of conn->want are converted to
fuse_{set,unset}_feature_flag().
Closes: https://github.com/libfuse/libfuse/issues/1171
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
(cherry picked from commit f68970cd235a7e14026ca0f6240428bbebe8223b)
|
|
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
The code uses std::min() which expects its arguments to be size_t. Two
times it uses an offset declared as off_t. While both size_t and off_t
are 32-bit integers, the latter is signed. On 64 bit architectures
the conversation of off_t -> size_t performed automatically. On 32 bit
architectures it needs a type cast.
Signed-off-by: Laszlo Boszormenyi (GCS) <gcs@debian.org>
|
|
Cleaning... 18 files.
[35/69] Compiling C object example/passthrough_ll.p/passthrough_ll.c.o
../example/passthrough_ll.c: In function ‘lo_opendir’:
../example/passthrough_ll.c:666:20: warning: ‘fd’ may be used uninitialized [-Wmaybe-uninitialized]
666 | if (fd != -1)
| ^
../example/passthrough_ll.c:637:13: note: ‘fd’ was declared here
637 | int fd;
| ^~
[38/69] Compiling C object test/test_syscalls.p/test_syscalls.c.o
../test/test_syscalls.c: In function ‘test_seekdir’:
../test/test_syscalls.c:804:16: warning: ‘de’ may be used uninitialized [-Wmaybe-uninitialized]
804 | while (de)
| ^~
../test/test_syscalls.c:776:24: note: ‘de’ was declared here
776 | struct dirent *de;
| ^~
These are actually valid.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
passthrough example supports the --plus command line argument to reply
to readdirplus with fill_dir_plus and unspecified (0) fill offsets.
As explained in this comment:
https://github.com/libfuse/libfuse/pull/896#issuecomment-1978917041
passthrough example needs a few more changes to be able to test commit
dd95d13a ("fix readdirplus when filler is called with zero offset (#896))
With the changes in this commit, readdirplus without fill offsets
can be tested to verify the readdirplus fix above with command line:
passthrough --plus -o auto_cache,modules=subdir,subdir=/src /mnt
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
After pulling latest code, memfs_ll build would fail because it
builds with C++11.
Changing the default of cpp_std in meson.build is not enough to fix this
problem even if user runs 'meson setup --reconfigure'.
I had to run 'meson setup -Dcpp_std= --reconfigure' to fix the build
as mentioned in this meson issue:
https://github.com/mesonbuild/meson/issues/8062#issuecomment-1568249672
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
In cache=always mode, set keep_cache flag in opendir(), same as done
in open() and same as passthrough_hp does in opendir().
In the default cache=auto mode, use readdir cache, but do not set
keep_cache, same as regular files use page cache for an open file,
but do not keep_cache for a new open.
Note that passthrough_hp by default behaves the same as passthrough_ll
cache=always mode and supports the cache=never mode with --nocache, but
it does not support the equivalent of cache=auto mode.
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
The previous fields are left for ABI compatibility, although
it is not beautiful to add that complexity when we have to increase
the so-version as we had ABI breakage anyway.
example/printcap is simplified to use an array, as every line would
have needed to be modified anyway. Missing 'FUSE_CAP_PASSTHROUGH'
was added.
Signed-off-by: Bernd Schubert <bernd@bsbernd.com>
|
|
This is an initial implementation and quickly hacked together
within a few hours - issues expected.
This also increase to C++17 as memfs_ll makes use of more
recent features.
Background to create this was actually to be able to test large
file names (3 * 1024B), which couldn't be achieved with passthrough
file system as non of the underlying file systems seems to support
that.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Note that name hashes and using paths as parameters
makes it very hard to support
anonymous files in the high level API.
Known Issues:
- tests have to bail out when O_TMPFILE is not supported.
This will always be the case with high level passthrough implementations.
- test_create_and_link_tmpfile has to be skipped
due to unidentified problems with github runner
|
|
This should stop some more xfstest test failures.
|
|
Simplify setting feature flags a bit by adding a helper
function.
Also move the check for valid flags into a funtion we can re-use
in a later patch.
|
|
fuse_loop_cfg_set_idle_threads() was by accident and
setting it might cause a performance issue.
|
|
These flags should be also set for O_CREAT
|
|
sfs_open and sfs_create set fi->direct_io (FOPEN_DIRECT_IO) when
O_DIRECT is given, in order to benefit from a shared inode lock
in kernel, i.e. to get parallel DIO writes. However, kernel side
disabled passthrough when FOPEN_DIRECT_IO is set. Reads/writes
had been totally failing in this case for O_DIRECT as
sfs_write_buf() and sfs_read() have a sanity check. That sanity
check could be modified, but for performance passthrough is
better than parallel DIO, hence, we only want automatic
FOPEN_DIRECT_IO for O_DIRECT when passthrough is not enabled.
Fixes: https://github.com/libfuse/libfuse/issues/1027
This also fixes automatically switching to FOPEN_DIRECT_IO
for O_DIRECT in sfs_create().
|
|
High level examples were already using it, but not
lowlevel. Also update the documentation.
|
|
fuse_config.h is used in libfuse internally. This may confuse developers
that fuse_config.h is used in the examples.
|
|
This is an addition to commit e75d2c54a347. This example sets
FUSE_USE_VERSION = 34 but uses fuse_loop_cfg_* APIs, which is
not allowed since these APIs are not introduced in version 34.
|
|
Fix some clang-tidy warnings.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
This is an addition to commit a8f1ae35af66, which
introduced the 312 API, but didn't set the right
API version in all examples.
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
Commit 170edc6a8ef0 added dot and dotdot (. and ..) to readdir
results, but introduced an issue when max number of entries
was reached - lookup count must not be decreased without
doing the lookup.
With ext4 as underlying file system readir seems to return . and ..
at random offsets and randomly failed xfstests for me.
This also fixes indentation, as passthrough_hp.cc does not follow
the linux indentation style (if we decide to fix this, it needs
to be done for the entire file).
Signed-off-by: Bernd Schubert <bschubert@ddn.com>
|
|
I see random ENOTCONN failures in xfstest generic/013 and generic/014
in my branch, but earliest on the 2nd run - takes ~12hours to get
the issue, but then there are no further information logged.
ENOTCONN points to a daemon crash - I need backtraces and a core dump.
This adds optional handling of fatal signals to print a core dump
and optional syslog logging with these new public functions:
fuse_set_fail_signal_handlers()
In addition to the existing fuse_set_signal_handlers(). This is not
enabled together with fuse_set_signal_handlers(), as it is change
in behavior and file systems might already have their own fatal
handlers.
fuse_log_enable_syslog
Print logs to syslog instead of stderr
fuse_log_close_syslog
Close syslog (for now just does closelog())
Code in fuse_signals.c is also updated, to be an array of signals,
and setting signal handlers is now down with a for-loop instead
of one hand coded set_one_signal_handler() per signal.
|
|
generic/401 fails currently because it checks that "." and ".." are
listed as directory entries.
Include "." and ".." as listed directory entries in passthrough_hp's
readdir implementation.
Signed-off by: Joanne Koong <joannelkoong@gmail.com>
|
|
Follow the same format as other arguments passed to example commands.
|
|
Notifications should be enabled here, but present command example would disable them.
|
|
should use EBADF instead
|
|
since there's no splice(2) support
|
|
generic/003 fails currently because if you specify -o relatime we will
fail to mount, but not return an error, so the test thinks that we
support relatime and then hilarity ensues. Set ret so that if we get
any failures while trying to mount we will properly error out.
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
|
|
The function fuse_session_process_buf_int() would do much things
for FUSE_INTERRUPT requests, even there are no FUSE_INTERRUPT requests:
1. check every non-FUSE_INTERRUPT request and add these requests to the
linked list(se->list) under a big lock(se->lock).
2. the function fuse_free_req() frees every request and remove them from
the linked list(se->list) under a bing lock(se->lock).
These operations are not meaningful when there are no FUSE_INTERRUPT requests,
and have a great impact on the performance of fuse filesystem because the big
lock for each request.
In some cases, FUSE_INTERRUPT requests are infrequent, even none at all.
Besides, the user-defined filesystem may do nothing for FUSE_INTERRUPT requests.
And the kernel side has the option "no_interrupt" in struct fuse_conn. This kernel option
can be enabled by return ENOSYS in libfuse for the reply of FUSE_INTERRUPT request.
But I don't find the code to enable the "no_interrupt" kernel option in libfuse.
So add the no_interrupt support, and when this operaion is enabled:
1. remove the useless locking operaions and list operations.
2. return ENOSYS for the reply of FUSE_INTERRUPT request to inform the kernel to disable
FUSE_INTERRUPT request.
|
|
Add support for filesystem passthrough read/write of files.
When the FUSE_PASSTHROUGH capability is enabled, the FUSE server may
decide, while handling the "open" or "create" requests, if the given
file can be accessed by that process in "passthrough" mode, meaning that
all the further read and write operations would be forwarded by the
kernel directly to the backing file rather than to the FUSE server.
All requests other than read or write are still handled by the server.
This allows for an improved performance on reads and writes, especially
in the case of reads at random offsets, for which no (readahead)
caching mechanism would help, reducing the performance gap between FUSE
and native filesystem access.
Extend also the passthrough_hp example with the new passthrough feature.
This example opens a kernel backing file per FUSE inode on the first
FUSE file open of that inode and closes the backing file on the release
of the last FUSE file on that inode.
All opens of the same inode passthrough to the same backing file.
A combination of fi->direct_io and fi->passthrough is allowed.
It means that read/write operations go directly to the server, but mmap
is done on the backing file.
This allows to open some fds of the inode in passthrough mode and some
fd of the same inode in direct_io/passthrough_mmap mode.
Signed-off-by: Alessio Balsini <balsini@android.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
|
|
Convert all the remaining users of fuse_session_loop_mt() to
the new 3.12 config api.
|
|
By virtio-fs and libfuse fuse_custom_io, passthrough_ll could be
a virtio filesystem device backend, this bug was found when doing
mount, fsstress and umount repeatedly.
Signed-off-by: Xiaoguang Wang <lege.wang@jaguarmicro.com>
Co-authored-by: Xiaoguang Wang <lege.wang@jaguarmicro.com>
|
|
Add more documentation for FUSE_CAP_EXPORT_SUPPORT
Also remove the flag from passthrough_ll.c and passthrough_hp.cc
as these implementations do _not_ handle that flag. They just
cast fuse_ino_t to an inode and cause a heap buffer overflow
for unknown objects (simplest reproducer are the examples
in "man 2 open_by_handle_at", but to unmount/mount the file
system after name_to_handle_at and before open_by_handle_at).
Fixes https://github.com/libfuse/libfuse/issues/838
---------
Co-authored-by: Nikolaus Rath <Nikolaus@rath.org>
|
|
This just adds in the basic handler, but does not
use it yet in examples.
|
|
In order to use the fuse_fill_dir_t function in a C++ program, add the enum item:
FUSE_FILL_DIR_DEFAULTS
Without this change g++ compilation failed with
example/hello.c:94:35: error: invalid conversion from ‘int’ to ‘fuse_fill_dir_flags’ [-fpermissive]
94 | filler(buf, ".", NULL, 0, 0);
| ^
| |
| int
|
|
Similar issue as fixed in commit
3c7ba570 "examples/notify_store_retrieve: Add a clean shutdown".
Basically a clean shutdown was missing, but even with clean
shutdown it does not work, as kernel side releases inodes before
sending FUSE_DESTROY - the intervaled thread then gets
-ENOENT.
Co-authored-by: Bernd Schubert <bschubert@ddn.com>
|
|
As noticed by valgrind in issue #907 example/poll.c
triggers a use-after-free
==85200== Thread 2:
==85200== Invalid read of size 4
==85200== at 0x485E54A: send_notify_iov (fuse_lowlevel.c:2267)
==85200== by 0x485E54A: fuse_lowlevel_notify_poll (fuse_lowlevel.c:2289)
==85200== by 0x1096F2: fsel_producer (poll.c:245)
==85200== by 0x4897EA6: start_thread (pthread_create.c:477)
==85200== by 0x49ADA6E: clone (clone.S:95)
==85200== Address 0x5291d68 is 392 bytes inside a block of size 920 free'd
==85200== at 0x48399AB: free (vg_replace_malloc.c:538)
==85200== by 0x485A12C: fuse_destroy (fuse.c:5103)
==85200== by 0x486220F: fuse_main_real (helper.c:389)
==85200== by 0x1091D6: main (poll.c:288)
==85200== Block was alloc'd at
==85200== at 0x483AB65: calloc (vg_replace_malloc.c:760)
==85200== by 0x485BAA0: fuse_session_new (fuse_lowlevel.c:3036)
==85200== by 0x4859AF2: fuse_new@@FUSE_3.1 (fuse.c:4966)
==85200== by 0x4862129: fuse_main_real (helper.c:345)
==85200== by 0x1091D6: main (poll.c:288)
Issue is that the "fsel_producer" thread is still active after
fuse_destroy - it gets destructed too late.
|
|
This test was racy, the lookup counter must only be increased once
kernel side has lookup completed and knows about the inode.
However, this is still racy as
app thread
(python script) kernel-side libfuse thread kernel
----------------------------------------------------------------------
open file
lookup req
wait
handle req
reply
wake app thread
return
new_inode()
<continue file open>
So actually on libfuse side even after returning from kernel side
it is not ensured that the kernel has created the inode. I.e.
using lookup_cnt in the test is still racy.
A new variabled 'open_cnt' is added that is only increased in open
Using open_cnt should be safe to use as kernel side (with atomic-open)
first does lookup, from that data creates the inode and only then sends
an open request. I.e. update_fs_loop() must only call
fuse_lowlevel_notify_store() once it is absolutely sure kernel side has
created the inode (open_cnt) and when it is sure the kernel inode still
exists (lookup_cnt).
Not really nice, but we actually need to accept some errors, as
these still come up at umount time. Typically it is hard to hit,
but tests in github actually frequently get it. Actually, it can be
easily reproduced by commenting out the sleep line in
update_fs_loop(). Underlying issue is that kernel side is
sending ->destroy() only when it already internally released all
inodes - too late for this test. The errors I run into are ENOENT
and EBADFD, but I added back in ENODEV for safety.
In order to avoid any other kind races mutex lock is also introduced.
|
|
Shared locks (parallel_direct_writes) cannot be enabled for O_DIRECT, as
O_DIRECT may be set past file open time with fcntl(fd, F_SETFD, ...).
Kernel side fuse has precautions for shared lock direct-IO (direct_io in
libfuse), as it needs an exclusive inode lock when direct and page cache
IO happend at the same time.
In order to enjoy the parallel_direct_writes feature (i.e., get a shared
lock, not exclusive lock) for writes to the same file), direct_io is needed.
The feature direct_io is corresponding to FOPEN_DIRECT_IO in fuse kernel.
FOPEN_DIRECT_IO and O_DIRECT are not entirely the same as described above.
So enable direct_io (i.e., FOPEN_DIRECT_IO in fuse kernel) to enjoy parallel
direct_writes.
Some patches related to FOPEN_DIRECT_IO and O_DIRECT are below:
https://lore.kernel.org/all/753d6823-e984-4730-a126-d66b65ea772c@ddn.com
|
|
Move the parallel_direct_writes enable action to the init function in high level API,
it is more recommended just like commit 8ee553dac. Besides, add some comments to show
that the feature parallel_direct_writes is depend on the feature direct_io (refer to
kernel side patch series to consolidate direct IO, link: https://lwn.net/ml/linux-fsdevel/
20230918150313.3845114-1-bschubert@ddn.com for the reason).
|
|
This is not called FUSE_CAP_DIRECT_IO_RELAX, as the kernel flag
FUSE_DIRECT_IO_RELAX is supposed to be renamed to
FUSE_DIRECT_IO_ALLOW_MMAP. The corresponding kernel patches just
did not land yet.
|
|
On shutdown update_fs_loop() was failing frequently in github
test with ENOENT, but only ENODEV was tested for and only for the.
With clean shutdown there is no need to test for such errors at all.
|
|
|
|
The clone-fd option is set valued but not used in the context. Use it in the code.
|
|
All these passthrough examples don't need writes to be serialized.
Actually, most file systems probably handle non serialized parallel
direct writes - the FOPEN_PARALLEL_DIRECT_WRITES flag is just
to avoid a regression for those file system that rely on serialized
DIO writes in fuse kernel. Passthrough file system forward the IO
to another file system, which actually handles that internally -
serialized in fuser kernel is not needed.
|
|
|
|
Actually one had to use --clone-fd=1 instead of
just --clone-fd.
|
