| Age | Commit message (Collapse) | Author |
|---|
|
The issue happens when a rename causes a split in the destination pair.
If the destination pair is the same as the source pair, this triggers the
logic to keep both pairs in sync. Unfortunately, this logic didn't work,
because the source entry still resides in the old source pair, unlike
the destination pair, which is now in the new pair created by the split.
The best fix for now is to refetch the source pair after the changes to the
destination pair. This isn't the most efficient solution, but fortunately
this bug has already been fixed in the revamped move logic in littlefs v2
(currently in progress).
Found by ohoc
|
|
|
|
- Fixed shadowed variable warnings in lfs_dir_find.
- Fixed unused parameter warnings when LFS_NO_MALLOC is enabled.
- Added extra warning flags to CFLAGS.
- Updated tests so they don't shadow the "size" variable for -Wshadow
|
|
Paths such as the following were causing issues:
/tea/hottea/.
/tea/hottea/..
Unfortunately the existing structure for path lookup didn't make it very
easy to introduce proper handling in this case without duplicating the
entire skip logic for paths. So the lfs_dir_find function had to be
restructured a bit.
One odd side-effect of this is that now lfs_dir_find includes the
initial fetch operation. This kinda breaks the fetch -> op pattern of
the dir functions, but does come with a nice code size reduction.
|
|
One of the big simplifications in littlefs's implementation is the
complete lack of tracking free blocks, allowing operations to simply
drop blocks that are no longer in use.
However, this means the lookahead buffer can easily contain outdated
blocks that were previously deleted. This is usually fine, as littlefs
will rescan the storage if it can't find a free block in the lookahead
buffer, but after changes that caused littlefs to more conservatively
respect the alloc acks (e611cf5), any scanned blocks after an ack would
be incorrectly trusted.
The fix is to eagerly scan ahead in the lookahead when we allocate so
that alloc acks are better able to discredit old lookahead blocks. Since
usually alloc acks are tightly coupled to allocations of one or two blocks,
this allows littlefs to properly rescan every set of allocations.
This may still be a concern if there is a long series of worn out
blocks, but in the worst case littlefs will conservatively avoid using
blocks it's not sure about.
Found by davidefer
|
|
The name test_parallel gave off the incorrect impression that these
tests are multithreaded.
|
|
Like most of the lfs_dir_t functions, lfs_dir_append is responsible for
updating the lfs_dir_t struct if the underlying directory block is
moved. This property makes handling worn out blocks much easier by
removing the amount of state that needs to be considered during a
directory update.
However, extending the dir chain is a bit of a corner case. It's not
changing the old block, but callers of lfs_dir_append do assume the
"entry" will reside in "dir" after lfs_dir_append completes.
This issue only occurs when creating files, since mkdir does not use
the entry after lfs_dir_append. Unfortunately, the tests against
extending the directory chain were all made using mkdir.
Found by schouleu
|
|
Before this patch, when calling lfs_mkdir or lfs_file_open with root
as the target, littlefs wouldn't find the path properly and happily
run into undefined behaviour.
The fix is to populate a directory entry for root in the lfs_dir_find
function. As an added plus, this allowed several special cases around
root to be completely dropped.
|
|
Using gcc cross compilers and qemu:
- make test CC="arm-linux-gnueabi-gcc --static -mthumb" EXEC="qemu-arm"
- make test CC="powerpc-linux-gnu-gcc --static" EXEC="qemu-ppc"
- make test CC="mips-linux-gnu-gcc --static" EXEC="qemu-mips"
Also separated out Travis jobs and added some size reporting
|
|
Rather than tracking all in-flight blocks blocks during a lookahead,
littlefs uses an ack scheme to mark the first allocated block that
hasn't reached the disk yet. littlefs assumes all blocks since the
last ack are bad or in-flight, and uses this to know when it's out
of storage.
However, these unacked allocations were still being populated in the
lookahead buffer. If the whole block device fits in the lookahead
buffer, _and_ littlefs managed to scan around the whole storage while
an unacked block was still in-flight, it would assume the block was
free and misallocate it.
The fix is to only fill the lookahead buffer up to the last ack.
The internal free structure was restructured to simplify the runtime
calculation of lookahead size.
|
|
- Write on read-only file to return LFS_ERR_BADF
- Renaming directory onto file to return LFS_ERR_NOTEMPTY
- Changed LFS_ERR_INVAL in lfs_file_seek to assert
|
|
|
|
Flags used:
-Wall -Wextra -Wshadow -Wwrite-strings -Wundef -Wstrict-prototypes
-Wunused -Wunused-parameter -Wunused-function -Wunused-value
-Wmissing-prototypes -Wmissing-declarations -Wold-style-definition
|
|
- no previous prototype for ‘test_assert’
- no previous prototype for ‘test_count’
- unused parameter ‘b’ in test_count
- function declaration isn’t a prototype for main
|
|
The most useful part of -Werror is preventing code from being
merged that has warnings. However it is annoying for users who may have
different compilers with different warnings. Limiting -Werror to CI only
covers the main concern about warnings without limiting users.
|
|
Some of the tests were creating a variable `res`, however the test
system itself relies on it's own `res` variable. This worked out by
luck, but could lead to problems if the res variables were different
types.
Changed the generated variable in the test system to the less common
name `test`, which also works out to share the same prefix as other test
functions.
|
|
Found by user iamscottmoyers, this was an interesting bug with the test
system. If the new test.c file is generated fast enough, it may not have
a new timestamp and not get recompiled.
To fix, we can remove the specific files that need to be rebuilt (lfs and
test.o).
|
|
As a copy-on-write filesystem, the truncate function is a very nice
function to have, as it can take advantage of reusing the data already
written out to disk.
|
|
As noted by itayzafrir, removing a non-empty directory should
error with ENOTEMPTY, not EINVAL
|
|
In the open call, the LFS_O_TRUNC flag was correctly zeroing the file, but
it wasn't actually writing the change out to disk. This went unnoticed because
in the cases where the truncate was followed by a file write, the
updated contents would be written out correctly.
Marking the file as dirty if the file isn't already truncated fixes the
problem with the least impact. Also added better test cases around
truncating files.
|
|
This bug was a result of an annoying corner case around intermingling
signed and unsigned offsets. The boundary check that prevents seeking
a file to a position before the file was preventing valid seeks with
positive offsets.
This corner case is a bit more complicated than it looks because the
offset is signed, while the size of the file is unsigned. Simply
casting both to signed or unsigned offsets won't handle large files.
|
|
As it was, if a user operated on a directory while at the same
time iterating over the directory, the directory objects could
fall out of sync. In the best case, files may be skipped while
removing everything in a file, in the worst case, a very poorly
timed directory relocate could be missed.
Simple fix is to add the same directory tracking that is currently
in use for files, at a small code+complexity cost.
|
|
Matches the standard EEXIST name found on most systems. Other than
this name, all other common constant names were consistent in this
manner.
|
|
Short story, files are no longer committed to directories during
file sync/close if the last write did not complete successfully.
This avoids a set of interesting user-experience issues related
to the end-of-life behaviour of the filesystem.
As a filesystem approaches end-of-life, the chances of running into
LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after
at the end of a devices life, it's likely that operating in these
conditions hasn't been tested thoroughly.
In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after
parts of the file have been written out. If the program simply continues
and closes the file, the file is written out half completed. Since
littlefs has a strong garuntee the prevents half-writes, it's unlikely
this state of the file would be expected.
To make things worse, since close is also responsible for memory
cleanup, it's actually _impossible_ to continue working as it was
without leaking memory.
By prevent the file commits, end-of-life behaviour should at least retain
a previous copy of the filesystem without any surprises.
|
|
This is only an issue in the weird case that are worn down block is
left in the odd state of not being able to change the data that resides
on the block. That being said, this does pop up often when simulating
wear on block devices.
Currently, directory commits checked if the write succeeded by crcing the
block to avoid the additional RAM cost for another buffer. However,
before this commit, directory commits just checked if the block crc was
valid, rather than comparing to the expected crc. This would usually
work, unless the block was stuck in a state with valid crc.
The fix is to simply compare with the expected crc to find errors.
|
|
The "move problem" has been present in littlefs for a while, but I haven't
come across a solution worth implementing for various reasons.
The problem is simple: how do we move directory entries across
directories atomically? Since multiple directory entries are involved,
we can't rely entirely on the atomic block updates. It ends up being
a bit of a puzzle.
To make the problem more complicated, any directory block update can
fail due to wear, and cause the directory block to need to be relocated.
This happens rarely, but brings a large number of corner cases.
---
The solution in this patch is simple:
1. Mark source as "moving"
2. Copy source to destination
3. Remove source
If littlefs ever runs into a "moving" entry, that means a power loss
occured during a move. Either the destination entry exists or it
doesn't. In this case we just search the entire filesystem for the
destination entry.
This is expensive, however the chance of a power loss during a move
is relatively low.
|
|
|
|
lfs_file_seek returned the _previous_ file offset on success, where
most standards return the _calculated_ offset on success.
This just falls into me not noticing a mistake, and shows why it's
always helpful to have a second set of eyes on code.
|
|
Moslty just a hole in testing. Dir blocks were not being
correctly collected when removing entries from very large
files due to forgetting about the tail-bit in the directory
block size. The test hole has now been filled.
Also added lfs_entry_size to avoid having to repeat that
expression since it is a bit ridiculous
|
|
- out-of-bound read results in eof
- out-of-bound write will fill missing area with zeros
The write behaviour matches expected posix behaviour, but was
under consideration for not being dropped, since littlefs does
not support holes, and support of out-of-band seeks adds complexity.
However, it turned out filling with zeros was trivial, and only
cost an extra 74 bytes of flash (0.48%).
|
|
- Added handling for root to lfs_stat
- Corrected lfs_dir_find to update path even on failures
- Added more checks for missing directories in path
|
|
This off-by-one error was caused by a slight difference between the
pos argument to lfs_index_find and lfs_index_extend. When pos is on
a block boundary, lfs_index_extend expects block to point before pos,
as it would when writing a file linearly. But when seeking to that
pos, the lfs_index_find to warm things up just supplies the block it
expects pos to be in.
Fixed the off-by-one error and added a test case for several of these
cold seek+writes.
|
|
Directories still consume two full erase blocks, but now only program
the exact on-disk region to store the directory contents. This results
in a decent improvement in the amount of data written and read to the
device when doing directory operations.
Calculating the checksum of dynamically sized data is surprisingly
tricky, since the size of the data could also contain errors. For the
littlefs, we can assume the data size must fit in an erase block.
If the data size is invalid, we can just treat the block as corrupted.
|
|
This provides a limited form of wear leveling. While wear is
not actually balanced across blocks, the filesystem can recover
from corrupted blocks and extend the lifetime of a device nearly
as much as dynamic wear leveling.
For use-cases where wear is important, it would be better to use
a full form of dynamic wear-leveling at the block level. (or
consider a logging filesystem).
Corrupted block handling was simply added on top of the existing
logic in place for the filesystem, so it's a bit more noodly than
it may have to be, but it gets the work done.
|
|
Conveniently we previously added a linked-list of files
for things like this. This should handle most of the corner
cases where files are open during strange operations.
This also brings up the point that we aren't doing anything similar
for directories and don't even have a dir linked-list. After thinking
about it for a while, I've decided to leave out this handling for dirs.
It will likely be very complicated, with little gains as directories
are less used in embedded systems. Additionally, dirs are only open
for reading, and corruption will probably just cause the dir iteration
to terminate. If needed, correct handling of open directories can be
added later.
|
|
Also added better testing specifically for these corner cases
around the end of storage
|
|
Needed primarily for tracking block allocations, unfortunately
this prevents the freedom for the user to bitwise copy files.
|
|
Originally had two seperate positions for reading/writing,
but this is inconsistent with the the posix standard, which
has a single position for reading and writing.
Also added proper handling of when the file is dirty, just
added an internal flag for this state.
Also moved the entry out of the file struct, and rearranged
some members to clean things up.
|
|
Now matches the commonly used errno codes in name with the value
encoded as the negative errno code
|
|
A rather involved upgrade for both files and directories, seek and
related functions are now completely supported:
- lfs_file_seek
- lfs_file_tell
- lfs_file_rewind
- lfs_file_size
- lfs_dir_seek
- lfs_dir_tell
- lfs_dir_rewind
This change also highlighted the concern that lfs_off_t is unsigned,
whereas off_t is traditionally signed. Unfortunately, lfs_off_t is
already used intensively through the codebase, so in focusing on
moving forward and avoiding getting bogged down by details, I'm going to
keep it as is and use the signed type lfs_soff_t where necessary.
|
|
Removed scanning for stride
- Adds complexity with questionable benefit
- Can be added as an optimization later
Fixed handling around device boundaries and where lookahead may not be a
factor of the device size (consider small devices with only a few
blocks)
Added support for configuration with optional dynamic memory as found in
the caching configuration
|
|
This adds caching of the most recent read/program blocks, allowing
support of devices that don't have byte-level read+writes, along
with reduced device access on devices that do support byte-level
read+writes.
Note: The current implementation is a bit eager to drop caches where
it simplifies the cache layer. This layer is already complex enough.
Note: It may be worthwhile to add a compile switch for caching to
reduce code size, note sure.
Note: This does add a dependency on malloc, which could have a porting
layer, but I'm just using the functions from stdlib for now. These can be
overwritten with noops if the user controls the system, and keeps things
simple for now.
|
|
Before, the lfs had multiple paths to determine config options:
- lfs_config struct passed during initialization
- lfs_bd_info struct passed during block device initialization
- compile time options
This allowed different developers to provide their own needs
to the filesystem, such as the block device capabilities and
the higher level user's own tweaks.
However, this comes with additional complexity and action required
when the configurations are incompatible.
For now, this has been reduced to all information (including block
device function pointers) being passed through the lfs_config struct.
We just defer more complicated handling of configuration options to
the top level user.
This simplifies configuration handling and gives the top level user
the responsibility to handle configuration, which they probably would
have wanted to do anyways.
|
|
After quite a bit of prototyping, settled on the following functions:
- lfs_dir_alloc - create a new dir
- lfs_dir_fetch - load and check a dir pair from disk
- lfs_dir_commit - save a dir pair to disk
- lfs_dir_shift - shrink a dir pair to disk
- lfs_dir_append - add a dir entry, creating dirs if needed
- lfs_dir_remove - remove a dir entry, dropping dirs if needed
Additionally, followed through with a few other tweaks
|
|
|
|
No longer need to be stored on disk, can be simulated on
the chip side. As mentioned in other commits, the parent
entries had dozens of problems with atomic updates, as
well as making everything just a bit more complex than
is needed.
|
|
Removing the dependency to the parent pointer solves
many issues with non-atomic updates of children's
parent pointers with respect to any move operations.
However, this comes with an embarrassingly terrible
runtime as the only other option is to exhaustively
check every dir entry to find a child's parent.
Fortunately, deorphaning should be a relatively rare
operation.
|
|
Unfortunately, threading all dir blocks in a linked-list did
not come without problems.
While it's possible to atomically add a dir to the linked list
(by adding the new dir into the linked-list position immediately
after it's parent, requiring only one atomic update to the parent
block), it is not easy to make sure the linked-list is in a state
that always allows atomic removal of dirs.
The simple solution is to allow this non-atomic removal, with an
additional step to remove any orphans that could have been created
by a power-loss. This deorphan step is only run if the normal
allocator has failed.
|
|
In writing the initial allocator, I ran into the rather
difficult problem of trying to iterate through the entire
filesystem cheaply and with only constant memory consumption
(which prohibits recursive functions).
The solution was to simply thread all directory blocks onto a
massive linked-list that spans the entire filesystem.
With the linked-list it was easy to create a traverse function
for all blocks in use on the filesystem (which has potential
for other utility), and add the rudimentary block allocator
using a bit-vector.
While the linked-list may add complexity (especially where
needing to maintain atomic operations), the linked-list helps
simplify what is currently the most expensive operation in
the filesystem, with no cost to space (the linked-list can
reuse the pointers used for chained directory blocks).
|
|
All path iteration all goes through the lfs_dir_find function,
which manages the syntax of paths and updates the path pointer
to just the name stored in the dir entry.
Also added directory chaining, which allows more than one block
per directory. This is a simple linked list.
|
