Ported directory_handle for POSIX from AFIO v1.

17 files changed, 518 insertions, 79 deletions

diff --git a/Readme.md b/Readme.md
index 567db023..b8eaed39 100644
--- a/Readme.md
+++ b/Readme.md
@@ -13,10 +13,13 @@ throws, have it detect __cpp_exceptions and skip those implementations.
 - [ ] Move handle caching into native_handle_type? Overlapped flag is especially needed.
 - [ ] Implement the long planned ACID key-value BLOB store
 with a very simple engine based on atomic renames and send it to Boost for peer review.
+  - You may need compression, https://github.com/johnezang/pithy looks easily convertible
+into header-only C++ and has a snappy-like performance to compression ratio. Make sure
+you merge the bug fixes from the forks first.
 
 - [ ] All time based kernel tests need to use soak test based API and auto adjust to
 valgrind.
-- [ ] Raise the sanitisers on per-commit CI via ctest.
+- [x] Raise the sanitisers on per-commit CI via ctest.
 - [ ] Rename all ParseProjectVersionFromHpp etc to parse_project_version_from_hpp etc
 - [ ] In DEBUG builds, have io_handle always not fill buffers passed to remind
 people to use pointers returned!
@@ -87,8 +90,8 @@ handles on some storage i.e. storage needs to be kept in a global map.
   - [ ] Should have decile bucketing e.g. percentage in bottom 10%, percentage
   in next 10% etc. Plus mean and stddev.
   - [ ] Should either be resettable or subtractable i.e. points can be diffed.
-  - [ ] Add IOPS QD=1..N storage profile test
-  - [ ] Add throughput storage profile test
+  - [x] Add IOPS QD=1..N storage profile test
+  - [x] Add throughput storage profile test
 - [ ] Output into YAML comparable hashes for OS + device + FS + flags
 so we can merge partial results for some combo into the results database.
 - [ ] Write YAML parsing tool which merges fs_probe_results.yaml into
@@ -99,13 +102,16 @@ is named FS + device e.g.
 
 ### Algorithms library `AFIO_V2_NAMESPACE::algorithm` todo:
 - [ ] Add an intelligent on demand memory mapper:
-  - Use one-two-three level page system, so 4Kb/2Mb/?. Files under 2Mb need just
+  - Use one-two-three level page system, so 4Kb/2Mb/1Gb. Files under 2Mb need just
 one indirection.
   - Page tables need to also live in a potentially mapped file
   - Could speculatively map 4Kb chunks lazily and keep an internal map of 4Kb
 offsets to map. This allows more optimal handing of growing files.
   - WOULD BE NICE: Copy on Write support which collates a list of dirtied 4Kb
 pages and could write those out as a delta.
+    - Perhaps Snappy compression could be useful? It is continuable from a base
+if you dump out the dictionary i.e. 1Mb data compressed, then add 4Kb delta, you can
+compress the additional 4Kb very quickly using the dictionary from the 1Mb.
     - LATER: Use guard pages to toggle dirty flag per initial COW
 - [ ] Store in EA or a file called .spookyhashes or .spookyhash the 128 bit hash of
 a file and the time it was calculated. This can save lots of hashing work later.
@@ -138,6 +144,40 @@ time but saving storage where possible.
 are the same inode.
 
 
+### Eventual transactional key-blob store:
+- What's the least possible complex implementation based on files and directories?
+  - `store/index` is 48 bit counter + open hash map of 128 bit key to blob
+  identifier (64 bits) which can be:
+  
+    1. `store/small/01-3f` for blobs < 65520 bytes. Each blob is padded to 64 byte
+    multiple and tail record with 6 byte (48 bit) counter + 2 byte size aligned
+    at end + optional hash. There are 15 of these used to maximise write concurrency.
+    Blob identifier is top 6 bits smallfile id, cannot be 0.
+    Remaining 58 bits is the offset into the smallfile (shifted left 6 bits, all records in smallfiles are at 64 byte multiples).
+    
+    2. `store/large/*` for blobs >= 65520.
+      - `store/large/hexkey/48bithexcounter` stores each blob
+      - Last 64 bytes contains magic, size, optional hash.
+    Blob identifier is top 6 bits zero. Next 10 bits is 4 bits mantissa shifted left
+    6 bits of shift (0-63) for approx size. Remaining 48 bits is counter.
+    
+  - `store/config` keeps:
+    - transactions enabled or not.
+    - write concurrency (i.e. number of small files)
+    - mmap enable or not (i.e. can be used over network drive)
+    - content hash used e.g. SpookyHash
+    - compression used e.g. pithy
+    - dirty flag i.e. do fsck on next first open
+      - `O_SYNC` was on or not last open (affects severity of any fsck).
+    - shared lock kept on config so we know last user exit/first user enter
+- How do I detect transaction conflicts?
+  - Could lock the bottom 64 bits of key hashes being updated during write?
+  False abort between colliding keys possible though.
+  - Could lock the 48 bit counter for the revision of each blob we are basing
+  this transaction on? No false abort, but instead updates of keys
+  related in last transaction can remove concurrency.
+
+
 ## Commits and tags in this git repository can be verified using:
 <pre>
 -----BEGIN PGP PUBLIC KEY BLOCK-----
diff --git a/include/afio/outcome b/include/afio/outcome
-Subproject a7e4195fc752756032d6968c0077cd3c1a54f11
+Subproject 3b55da491459d728ff5f0eff6bc4bd4904f117b
diff --git a/include/afio/quickcpplib b/include/afio/quickcpplib
-Subproject b271c38263ed2202f093b918ce4a1081027df3e
+Subproject 68280e4497c002bacb2c45f42db9aab9a038cfb
diff --git a/include/afio/revision.hpp b/include/afio/revision.hpp
index 61454952..c9068af8 100644
--- a/include/afio/revision.hpp
+++ b/include/afio/revision.hpp
@@ -1,4 +1,4 @@
 // Note the second line of this file must ALWAYS be the git SHA, third line ALWAYS the git SHA update time
-#define AFIO_PREVIOUS_COMMIT_REF    1e256c3ec9f9dc50a38785536ff1f330e3807943
-#define AFIO_PREVIOUS_COMMIT_DATE   "2017-08-11 16:26:47 +00:00"
-#define AFIO_PREVIOUS_COMMIT_UNIQUE 1e256c3e
+#define AFIO_PREVIOUS_COMMIT_REF    6e868b4ea5b406c1673403b8c828cd0c76a70bbb
+#define AFIO_PREVIOUS_COMMIT_DATE   "2017-08-13 01:40:46 +00:00"
+#define AFIO_PREVIOUS_COMMIT_UNIQUE 6e868b4e
diff --git a/include/afio/v2.0/config.hpp b/include/afio/v2.0/config.hpp
index 3f3d0b02..b12bb2cd 100644
--- a/include/afio/v2.0/config.hpp
+++ b/include/afio/v2.0/config.hpp
@@ -65,12 +65,6 @@ Distributed under the Boost Software License, Version 1.0.
 #endif
 #endif
 
-#if !defined(AFIO_DISABLE_RACE_FREE_PATH_FUNCTIONS)
-#ifdef __APPLE__
-#define AFIO_DISABLE_RACE_FREE_PATH_FUNCTIONS 1
-#endif
-#endif
-
 
 #if defined(_WIN32)
 #if !defined(_UNICODE)
@@ -102,7 +96,7 @@ Distributed under the Boost Software License, Version 1.0.
 #ifndef __cpp_variadic_templates
 #error AFIO needs variadic template support in the compiler
 #endif
-#if __cpp_constexpr < 201304
+#if __cpp_constexpr < 201304L && !defined(_MSC_VER)
 #error AFIO needs relaxed constexpr (C++ 14) support in the compiler
 #endif
 #ifndef __cpp_init_captures
diff --git a/include/afio/v2.0/detail/impl/posix/directory_handle.ipp b/include/afio/v2.0/detail/impl/posix/directory_handle.ipp
new file mode 100644
index 00000000..022b95e0
--- /dev/null
+++ b/include/afio/v2.0/detail/impl/posix/directory_handle.ipp
@@ -0,0 +1,321 @@
+/* A handle to a directory
+(C) 2017 Niall Douglas <http://www.nedproductions.biz/> (20 commits)
+File Created: Aug 2017
+
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License in the accompanying file
+Licence.txt or at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+
+Distributed under the Boost Software License, Version 1.0.
+(See accompanying file Licence.txt or copy at
+http://www.boost.org/LICENSE_1_0.txt)
+*/
+
+#include "../../../directory_handle.hpp"
+#include "import.hpp"
+
+#ifdef QUICKCPPLIB_ENABLE_VALGRIND
+#include "../quickcpplib/valgrind/memcheck.h"
+#define AFIO_VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE(a, b) VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE((a), (b))
+#else
+#define AFIO_VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE(a, b)
+#endif
+
+#include <dirent.h> /* Defines DT_* constants */
+#include <fnmatch.h>
+#include <sys/stat.h>
+#include <sys/syscall.h>
+
+AFIO_V2_NAMESPACE_BEGIN
+
+result<directory_handle> directory_handle::directory(const path_handle &base, path_view_type path, mode _mode, creation _creation, caching _caching, flag flags) noexcept
+{
+  result<directory_handle> ret(directory_handle(native_handle_type(), 0, 0, _caching, flags));
+  native_handle_type &nativeh = ret.value()._v;
+  AFIO_LOG_FUNCTION_CALL(&ret);
+  if(_creation == creation::truncate)
+    return std::errc::operation_not_supported;
+  nativeh.behaviour |= native_handle_type::disposition::directory;
+  OUTCOME_TRY(attribs, attribs_from_handle_mode_caching_and_flags(nativeh, _mode, _creation, _caching, flags));
+#ifdef O_DIRECTORY
+  attribs |= O_DIRECTORY;
+#endif
+#ifdef O_SEARCH
+  attribs |= O_SEARCH;
+#endif
+  path_view::c_str zpath(path);
+  if(base.is_valid())
+  {
+#ifdef AFIO_DISABLE_RACE_FREE_PATH_FUNCTIONS
+    return std::errc::function_not_supported;
+#else
+    if(_creation == creation::if_needed || _creation == creation::only_if_not_exist)
+    {
+      if(-1 == ::mkdirat(base.native_handle().fd, zpath.buffer, 0x1f8 /*770*/))
+      {
+        if(EEXIST != errno || _creation == creation::only_if_not_exist)
+        {
+          return {errno, std::system_category()};
+        }
+      }
+      attribs &= ~(O_CREAT | O_EXCL);
+    }
+    nativeh.fd = ::openat(base.native_handle().fd, zpath.buffer, attribs);
+#endif
+  }
+  else
+  {
+    if(_creation == creation::if_needed || _creation == creation::only_if_not_exist)
+    {
+      if(-1 == ::mkdir(zpath.buffer, 0x1f8 /*770*/))
+      {
+        if(EEXIST != errno || _creation == creation::only_if_not_exist)
+        {
+          return {errno, std::system_category()};
+        }
+      }
+      attribs &= ~(O_CREAT | O_EXCL);
+    }
+    nativeh.fd = ::open(zpath.buffer, attribs);
+  }
+  if(-1 == nativeh.fd)
+  {
+    return {errno, std::system_category()};
+  }
+  if(!(flags & flag::disable_safety_unlinks))
+  {
+    if(!ret.value()._fetch_inode())
+    {
+      // If fetching inode failed e.g. were opening device, disable safety unlinks
+      ret.value()._flags &= ~flag::disable_safety_unlinks;
+    }
+  }
+  if(ret.value().are_safety_fsyncs_issued())
+    fsync(nativeh.fd);
+  return ret;
+}
+
+result<directory_handle> directory_handle::clone() const noexcept
+{
+  AFIO_LOG_FUNCTION_CALL(this);
+  result<directory_handle> ret(directory_handle(native_handle_type(), _devid, _inode, _caching, _flags));
+  ret.value()._v.behaviour = _v.behaviour;
+  ret.value()._v.fd = ::dup(_v.fd);
+  if(-1 == ret.value()._v.fd)
+    return {errno, std::system_category()};
+  return ret;
+}
+
+result<directory_handle::enumerate_info> directory_handle::enumerate(buffers_type &tofill, path_view_type glob, filter /*unused*/, span<char> kernelbuffer) const noexcept
+{
+  AFIO_LOG_FUNCTION_CALL(this);
+  if(tofill.empty())
+    return enumerate_info{stat_t::want::none, false};
+  // Is glob a single entry match? If so, this is really a stat call
+  path_view_type::c_str zglob(glob);
+  if(!glob.empty() && !glob.contains_glob())
+  {
+    struct stat s;
+    if(-1 == ::fstatat(_v.fd, zglob.buffer, &s, AT_SYMLINK_NOFOLLOW))
+      return {errno, std::system_category()};
+    tofill[0].stat.st_dev = s.st_dev;
+    tofill[0].stat.st_ino = s.st_ino;
+    tofill[0].stat.st_type = to_st_type(s.st_mode);
+    tofill[0].stat.st_perms = s.st_mode & 0xfff;
+    tofill[0].stat.st_nlink = s.st_nlink;
+    tofill[0].stat.st_uid = s.st_uid;
+    tofill[0].stat.st_gid = s.st_gid;
+    tofill[0].stat.st_rdev = s.st_rdev;
+#ifdef __ANDROID__
+    tofill[0].stat.st_atim = to_timepoint(*((struct timespec *) &s.st_atime));
+    tofill[0].stat.st_mtim = to_timepoint(*((struct timespec *) &s.st_mtime));
+    tofill[0].stat.st_ctim = to_timepoint(*((struct timespec *) &s.st_ctime));
+#elif defined(__APPLE__)
+    tofill[0].stat.st_atim = to_timepoint(s.st_atimespec);
+    tofill[0].stat.st_mtim = to_timepoint(s.st_mtimespec);
+    tofill[0].stat.st_ctim = to_timepoint(s.st_ctimespec);
+#else  // Linux and BSD
+    tofill[0].stat.st_atim = to_timepoint(s.st_atim);
+    tofill[0].stat.st_mtim = to_timepoint(s.st_mtim);
+    tofill[0].stat.st_ctim = to_timepoint(s.st_ctim);
+#endif
+    tofill[0].stat.st_size = s.st_size;
+    tofill[0].stat.st_allocated = (handle::extent_type) s.st_blocks * 512;
+    tofill[0].stat.st_blocks = s.st_blocks;
+    tofill[0].stat.st_blksize = s.st_blksize;
+#ifdef HAVE_STAT_FLAGS
+    tofill[0].stat.st_flags = s.st_flags;
+#endif
+#ifdef HAVE_STAT_GEN
+    tofill[0].stat.st_gen = s.st_gen;
+#endif
+#ifdef HAVE_BIRTHTIMESPEC
+#if defined(__APPLE__)
+    tofill[0].stat.st_birthtim = to_timepoint(s.st_birthtimespec);
+#else
+    tofill[0].stat.st_birthtim = to_timepoint(s.st_birthtim);
+#endif
+#endif
+    tofill[0].stat.st_sparse = ((handle::extent_type) s.st_blocks * 512) < (handle::extent_type) s.st_size;
+    tofill._resize(1);
+    static constexpr stat_t::want default_stat_contents = stat_t::want::dev | stat_t::want::ino | stat_t::want::type | stat_t::want::perms | stat_t::want::nlink | stat_t::want::uid | stat_t::want::gid | stat_t::want::rdev | stat_t::want::atim | stat_t::want::mtim | stat_t::want::ctim | stat_t::want::size |
+                                                          stat_t::want::allocated | stat_t::want::blocks | stat_t::want::blksize
+#ifdef HAVE_STAT_FLAGS
+                                                          | stat_t::want::flags
+#endif
+#ifdef HAVE_STAT_GEN
+                                                          | stat_t::want::gen
+#endif
+#ifdef HAVE_BIRTHTIMESPEC
+                                                          | stat_t::want::birthtim
+#endif
+                                                          | stat_t::want::sparse;
+    return enumerate_info{default_stat_contents, true};
+  }
+#ifdef __linux__
+  // Unlike FreeBSD, Linux doesn't define a getdents() function, so we'll do that here.
+  typedef int (*getdents64_t)(int, char *, unsigned);
+  static getdents64_t getdents = (getdents64_t)[](int fd, char *buf, unsigned count)->int { return syscall(SYS_getdents64, fd, buf, count); };
+  typedef dirent64 dirent;
+#endif
+#ifdef __APPLE__
+  // OS X defines a getdirentries64() kernel syscall which can emulate getdents
+  typedef int (*getdents_emulation_t)(int, char *, unsigned);
+  static getdents_emulation_t getdents = (getdents_emulation_t)[](int fd, char *buf, unsigned count)->int
+  {
+    off_t foo;
+    return syscall(SYS_getdirentries64, fd, buf, count, &foo);
+  };
+#endif
+  if(!tofill._kernel_buffer && kernelbuffer.empty())
+  {
+    // Let's assume the average leafname will be 64 characters long.
+    size_t toallocate = (sizeof(dirent) + 64) * tofill.size();
+    char *mem = new(std::nothrow) char[toallocate];
+    if(!mem)
+    {
+      return std::errc::not_enough_memory;
+    }
+    tofill._kernel_buffer = std::unique_ptr<char[]>(mem);
+    tofill._kernel_buffer_size = toallocate;
+  }
+  stat_t::want default_stat_contents = stat_t::want::ino | stat_t::want::type;
+  dirent *buffer;
+  size_t bytesavailable;
+  int bytes;
+  bool done = false;
+  do
+  {
+    buffer = kernelbuffer.empty() ? (dirent *) tofill._kernel_buffer.get() : (dirent *) kernelbuffer.data();
+    bytesavailable = kernelbuffer.empty() ? tofill._kernel_buffer_size : kernelbuffer.size();
+// Seek to start
+#ifdef __linux__
+    if(-1 == ::lseek64(_v.fd, 0, SEEK_SET))
+      return {errno, std::system_category()};
+#else
+    if(-1 == ::lseek(_v.fd, 0, SEEK_SET))
+      return {errno, std::system_category()};
+#endif
+    bytes = getdents(_v.fd, (char *) buffer, bytesavailable);
+    if(kernelbuffer.empty() && bytes == -1 && EINVAL == errno)
+    {
+      tofill._kernel_buffer.reset();
+      size_t toallocate = tofill._kernel_buffer_size * 2;
+      char *mem = new(std::nothrow) char[toallocate];
+      if(!mem)
+      {
+        return std::errc::not_enough_memory;
+      }
+      tofill._kernel_buffer = std::unique_ptr<char[]>(mem);
+      tofill._kernel_buffer_size = toallocate;
+    }
+    else
+    {
+      done = true;
+    }
+  } while(!done);
+  if(bytes == 0)
+  {
+    tofill._resize(0);
+    return enumerate_info{default_stat_contents, true};
+  }
+  AFIO_VALGRIND_MAKE_MEM_DEFINED_IF_ADDRESSABLE(buffer, bytes);
+  size_t n = 0;
+  for(dirent *dent = buffer;; dent = (dirent *) ((size_t) dent + dent->d_reclen))
+  {
+    if((bytes -= dent->d_reclen) <= 0)
+    {
+      // Fill is complete
+      tofill._resize(n);
+      return enumerate_info{default_stat_contents, true};
+    }
+    if(!dent->d_ino)
+    {
+      continue;
+    }
+    size_t length = strchr(dent->d_name, 0) - dent->d_name;
+    if(length <= 2 && '.' == dent->d_name[0])
+    {
+      if(1 == length || '.' == dent->d_name[1])
+      {
+        continue;
+      }
+    }
+    if(!glob.empty() && fnmatch(zglob.buffer, dent->d_name, 0))
+    {
+      continue;
+    }
+    directory_entry &item = tofill[n];
+    item.leafname = path_view(dent->d_name, length);
+    item.stat = stat_t(nullptr);
+    item.stat.st_ino = dent->d_ino;
+    char d_type = dent->d_type;
+    switch(d_type)
+    {
+    case DT_BLK:
+      item.stat.st_type = filesystem::file_type::block;
+      break;
+    case DT_CHR:
+      item.stat.st_type = filesystem::file_type::character;
+      break;
+    case DT_DIR:
+      item.stat.st_type = filesystem::file_type::directory;
+      break;
+    case DT_FIFO:
+      item.stat.st_type = filesystem::file_type::fifo;
+      break;
+    case DT_LNK:
+      item.stat.st_type = filesystem::file_type::symlink;
+      break;
+    case DT_REG:
+      item.stat.st_type = filesystem::file_type::regular;
+      break;
+    case DT_SOCK:
+      item.stat.st_type = filesystem::file_type::socket;
+      break;
+    default:
+      // Don't say we return type
+      default_stat_contents = default_stat_contents & ~stat_t::want::type;
+      break;
+    }
+    n++;
+    if(n >= tofill.size())
+    {
+      // Fill is incomplete
+      return enumerate_info{default_stat_contents, false};
+    }
+  }
+}
+
+AFIO_V2_NAMESPACE_END
diff --git a/include/afio/v2.0/detail/impl/posix/file_handle.ipp b/include/afio/v2.0/detail/impl/posix/file_handle.ipp
index c9e4c9fd..8bf94740 100644
--- a/include/afio/v2.0/detail/impl/posix/file_handle.ipp
+++ b/include/afio/v2.0/detail/impl/posix/file_handle.ipp
@@ -150,6 +150,7 @@ result<file_handle> file_handle::temp_inode(path_view_type dirpath, mode _mode,
   nativeh.fd = ::open(zpath.buffer, attribs, 0600);
   if(-1 != nativeh.fd)
   {
+    ret.value()._flags |= flag::anonymous_inode;
     OUTCOME_TRYV(ret.value()._fetch_inode());  // It can be useful to know the inode of temporary inodes
     return ret;
   }
diff --git a/include/afio/v2.0/detail/impl/posix/fs_handle.ipp b/include/afio/v2.0/detail/impl/posix/fs_handle.ipp
index 0743431c..e77bb5f8 100644
--- a/include/afio/v2.0/detail/impl/posix/fs_handle.ipp
+++ b/include/afio/v2.0/detail/impl/posix/fs_handle.ipp
@@ -80,7 +80,7 @@ inline result<path_handle> containing_directory(optional<std::reference_wrapper<
         return success(std::move(currentdirh));
       // Open the same file name, and compare dev and inode
       path_view::c_str zpath(filename);
-      int fd = ::openat(currentdirh.native_handle().fd, filename.buffer, 0);
+      int fd = ::openat(currentdirh.native_handle().fd, zpath.buffer, 0);
       if(fd == -1)
         continue;
       auto unfd = undoer([fd] { ::close(fd); });
@@ -124,24 +124,41 @@ result<path_handle> fs_handle::parent_path_handle(deadline d) const noexcept
   return containing_directory({}, h, *this, d);
 }
 
-result<void> fs_handle::relink(const path_handle &base, path_view_type path, deadline d) noexcept
+result<void> fs_handle::relink(const path_handle &base, path_view_type path, bool atomic_replace, deadline d) noexcept
 {
   AFIO_LOG_FUNCTION_CALL(this);
-  auto &h = _get_handle();
+  auto &h = const_cast<handle &>(_get_handle());
   path_view::c_str zpath(path);
 #ifdef O_TMPFILE
   // If the handle was created with O_TMPFILE, we need a different approach
-  if(path.empty() && (h.kernel_caching() == handle::caching::temporary))
+  if(h.flags() & handle::flag::anonymous_inode)
   {
+    if(atomic_replace)
+      return std::errc::function_not_supported;
     char _path[PATH_MAX];
     snprintf(_path, PATH_MAX, "/proc/self/fd/%d", h.native_handle().fd);
     if(-1 == ::linkat(AT_FDCWD, _path, base.is_valid() ? base.native_handle().fd : AT_FDCWD, zpath.buffer, AT_SYMLINK_FOLLOW))
       return {errno, std::system_category()};
+    h._flags &= ~handle::flag::anonymous_inode;
+    return success();
   }
 #endif
   // Open our containing directory
   filesystem::path filename;
-  OUTCOME_TRY(dirh, containing_directory(filename, h, *this, d));
+  OUTCOME_TRY(dirh, containing_directory(std::ref(filename), h, *this, d));
+  if(!atomic_replace)
+  {
+// Some systems provide an extension for atomic non-replacing renames
+#ifdef RENAME_NOREPLACE
+    if(-1 != ::renameat2(dirh.native_handle().fd, filename.c_str(), base.is_valid() ? base.native_handle().fd : AT_FDCWD, zpath.buffer, RENAME_NOREPLACE))
+      return success();
+    if(EEXIST == errno)
+      return {errno, std::system_category()};
+#endif
+    // Otherwise we need to use linkat followed by renameat (non-atomic)
+    if(-1 == ::linkat(dirh.native_handle().fd, filename.c_str(), base.is_valid() ? base.native_handle().fd : AT_FDCWD, zpath.buffer, 0))
+      return {errno, std::system_category()};
+  }
   if(-1 == ::renameat(dirh.native_handle().fd, filename.c_str(), base.is_valid() ? base.native_handle().fd : AT_FDCWD, zpath.buffer))
     return {errno, std::system_category()};
   return success();
@@ -153,7 +170,7 @@ result<void> fs_handle::unlink(deadline d) noexcept
   auto &h = _get_handle();
   // Open our containing directory
   filesystem::path filename;
-  OUTCOME_TRY(dirh, containing_directory(filename, h, *this, d));
+  OUTCOME_TRY(dirh, containing_directory(std::ref(filename), h, *this, d));
   if(-1 == ::unlinkat(dirh.native_handle().fd, filename.c_str(), 0))
     return {errno, std::system_category()};
   return success();
diff --git a/include/afio/v2.0/detail/impl/posix/stat.ipp b/include/afio/v2.0/detail/impl/posix/stat.ipp
index c460b0f6..fa3261a9 100644
--- a/include/afio/v2.0/detail/impl/posix/stat.ipp
+++ b/include/afio/v2.0/detail/impl/posix/stat.ipp
@@ -33,24 +33,6 @@ static inline filesystem::file_type to_st_type(uint16_t mode)
 {
   switch(mode & S_IFMT)
   {
-#ifdef AFIO_USE_LEGACY_FILESYSTEM_SEMANTICS
-  case S_IFBLK:
-    return filesystem::file_type::block_file;
-  case S_IFCHR:
-    return filesystem::file_type::character_file;
-  case S_IFDIR:
-    return filesystem::file_type::directory_file;
-  case S_IFIFO:
-    return filesystem::file_type::fifo_file;
-  case S_IFLNK:
-    return filesystem::file_type::symlink_file;
-  case S_IFREG:
-    return filesystem::file_type::regular_file;
-  case S_IFSOCK:
-    return filesystem::file_type::socket_file;
-  default:
-    return filesystem::file_type::type_unknown;
-#else
   case S_IFBLK:
     return filesystem::file_type::block;
   case S_IFCHR:
@@ -67,7 +49,6 @@ static inline filesystem::file_type to_st_type(uint16_t mode)
     return filesystem::file_type::socket;
   default:
     return filesystem::file_type::unknown;
-#endif
   }
 }
 
@@ -90,7 +71,7 @@ AFIO_HEADERS_ONLY_MEMFUNC_SPEC result<size_t> stat_t::fill(const handle &h, stat
   size_t ret = 0;
 
   if(-1 == ::fstat(h.native_handle().fd, &s))
-    return { errno, std::system_category() };
+    return {errno, std::system_category()};
   if(wanted & want::dev)
   {
     st_dev = s.st_dev;
diff --git a/include/afio/v2.0/detail/impl/windows/directory_handle.ipp b/include/afio/v2.0/detail/impl/windows/directory_handle.ipp
index 9a583a54..cab6d35f 100644
--- a/include/afio/v2.0/detail/impl/windows/directory_handle.ipp
+++ b/include/afio/v2.0/detail/impl/windows/directory_handle.ipp
@@ -149,11 +149,14 @@ result<directory_handle> directory_handle::clone() const noexcept
   return ret;
 }
 
-result<bool> directory_handle::enumerate(buffers_type &tofill, path_view_type glob, filter filtering, span<char> kernelbuffer) const noexcept
+result<directory_handle::enumerate_info> directory_handle::enumerate(buffers_type &tofill, path_view_type glob, filter filtering, span<char> kernelbuffer) const noexcept
 {
+  static constexpr stat_t::want default_stat_contents = stat_t::want::ino | stat_t::want::type | stat_t::want::atim | stat_t::want::mtim | stat_t::want::ctim | stat_t::want::size | stat_t::want::allocated | stat_t::want::birthtim | stat_t::want::sparse | stat_t::want::compressed | stat_t::want::reparse_point;
   windows_nt_kernel::init();
   using namespace windows_nt_kernel;
   AFIO_LOG_FUNCTION_CALL(this);
+  if(tofill.empty())
+    return enumerate_info{stat_t::want::none, false};
   UNICODE_STRING _glob;
   memset(&_glob, 0, sizeof(_glob));
   path_view_type::c_str zglob(glob, true);
@@ -199,7 +202,9 @@ result<bool> directory_handle::enumerate(buffers_type &tofill, path_view_type gl
       tofill._kernel_buffer_size = toallocate;
     }
     else
+    {
       done = true;
+    }
   } while(!done);
   size_t n = 0;
   for(FILE_ID_FULL_DIR_INFORMATION *ffdi = buffer;; ffdi = (FILE_ID_FULL_DIR_INFORMATION *) ((size_t) ffdi + ffdi->NextEntryOffset))
@@ -208,7 +213,7 @@ result<bool> directory_handle::enumerate(buffers_type &tofill, path_view_type gl
     {
       // Fill is complete
       tofill._resize(n);
-      return true;
+      return enumerate_info{default_stat_contents, true};
     }
     size_t length = ffdi->FileNameLength / sizeof(wchar_t);
     if(length <= 2 && '.' == ffdi->FileName[0])
@@ -236,7 +241,7 @@ result<bool> directory_handle::enumerate(buffers_type &tofill, path_view_type gl
     if(n >= tofill.size())
     {
       // Fill is incomplete
-      return false;
+      return enumerate_info{default_stat_contents, false};
     }
   }
 }
diff --git a/include/afio/v2.0/detail/impl/windows/fs_handle.ipp b/include/afio/v2.0/detail/impl/windows/fs_handle.ipp
index 5280a6b2..98b870dc 100644
--- a/include/afio/v2.0/detail/impl/windows/fs_handle.ipp
+++ b/include/afio/v2.0/detail/impl/windows/fs_handle.ipp
@@ -106,7 +106,7 @@ result<path_handle> fs_handle::parent_path_handle(deadline d) const noexcept
   }
 }
 
-result<void> fs_handle::relink(const path_handle &base, path_view_type path, deadline d) noexcept
+result<void> fs_handle::relink(const path_handle &base, path_view_type path, bool atomic_replace, deadline d) noexcept
 {
   windows_nt_kernel::init();
   using namespace windows_nt_kernel;
@@ -145,7 +145,7 @@ result<void> fs_handle::relink(const path_handle &base, path_view_type path, dea
   IO_STATUS_BLOCK isb = make_iostatus();
   alignas(8) char buffer[sizeof(FILE_RENAME_INFORMATION) + 65536];
   FILE_RENAME_INFORMATION *fni = (FILE_RENAME_INFORMATION *) buffer;
-  fni->ReplaceIfExists = true;
+  fni->ReplaceIfExists = atomic_replace;
   fni->RootDirectory = base.is_valid() ? base.native_handle().h : nullptr;
   fni->FileNameLength = _path.Length;
   memcpy(fni->FileName, _path.Buffer, fni->FileNameLength);
diff --git a/include/afio/v2.0/directory_handle.hpp b/include/afio/v2.0/directory_handle.hpp
index cf237482..7637d346 100644
--- a/include/afio/v2.0/directory_handle.hpp
+++ b/include/afio/v2.0/directory_handle.hpp
@@ -47,12 +47,6 @@ struct directory_entry
   path_view leafname;
   //! The metadata retrieved for the directory entry
   stat_t stat;
-//! The default list of metadata retrieved by `enumerate()` on this platform
-#ifdef _WIN32
-  static constexpr stat_t::want default_stat_contents = stat_t::want::ino | stat_t::want::type | stat_t::want::atim | stat_t::want::mtim | stat_t::want::ctim | stat_t::want::size | stat_t::want::allocated | stat_t::want::birthtim | stat_t::want::sparse | stat_t::want::compressed | stat_t::want::reparse_point;
-#else
-  static constexpr stat_t::want default_stat_contents = stat_t::want::ino | stat_t::want::type;
-#endif
 };
 #ifndef NDEBUG
 // Is trivial in all ways, except default constructibility
@@ -210,10 +204,19 @@ public:
   */
   AFIO_HEADERS_ONLY_VIRTUAL_SPEC result<directory_handle> clone() const noexcept;
 
+  //! Completion information for `enumerate()`
+  struct enumerate_info
+  {
+    //! The list of stat metadata retrieved by `enumerate()` this call per `buffer_type`.
+    stat_t::want metadata;
+    //! Whether the directory was entirely read or not.
+    bool done;
+  };
   /*! Fill the buffers type with as many directory entries as will fit.
 
-  \return True if the entire directory was read into `tofill`, false otherwise.
-  `tofill`'s extent is adjusted to match the number of items read on exit.
+  \return Returns whether the entire directory was read into `tofill`, false otherwise,
+  and what metadata was filled in. `tofill`'s extent is adjusted to match the number of
+  items read on exit.
   \param tofill The buffers to fill.
   \param glob An optional shell glob by which to filter the items filled. Done kernel side on Windows, user side on POSIX.
   \param filtering Whether to filter out fake-deleted files on Windows or not.
@@ -225,10 +228,67 @@ public:
   If unset, at least one memory allocation, possibly more is performed.
   */
   AFIO_MAKE_FREE_FUNCTION
-  AFIO_HEADERS_ONLY_VIRTUAL_SPEC result<bool> enumerate(buffers_type &tofill, path_view_type glob = path_view_type(), filter filtering = filter::fastdeleted, span<char> kernelbuffer = span<char>()) const noexcept;
+  AFIO_HEADERS_ONLY_VIRTUAL_SPEC result<enumerate_info> enumerate(buffers_type &tofill, path_view_type glob = path_view_type(), filter filtering = filter::fastdeleted, span<char> kernelbuffer = span<char>()) const noexcept;
 };
 
 // BEGIN make_free_functions.py
+//! Swap with another instance
+inline void swap(directory_handle &self, directory_handle &o) noexcept
+{
+  return self.swap(std::forward<decltype(o)>(o));
+}
+/*! Create a handle opening access to a directory on path.
+
+\errors Any of the values POSIX open() or CreateFile() can return.
+*/
+inline result<directory_handle> directory(const path_handle &base, directory_handle::path_view_type _path, directory_handle::mode _mode = directory_handle::mode::read, directory_handle::creation _creation = directory_handle::creation::open_existing, directory_handle::caching _caching = directory_handle::caching::all,
+                                          directory_handle::flag flags = directory_handle::flag::none) noexcept
+{
+  return directory_handle::directory(std::forward<decltype(base)>(base), std::forward<decltype(_path)>(_path), std::forward<decltype(_mode)>(_mode), std::forward<decltype(_creation)>(_creation), std::forward<decltype(_caching)>(_caching), std::forward<decltype(flags)>(flags));
+}
+/*! Create a directory handle creating a randomly named file on a path.
+The file is opened exclusively with `creation::only_if_not_exist` so it
+will never collide with nor overwrite any existing entry.
+
+\errors Any of the values POSIX open() or CreateFile() can return.
+*/
+inline result<directory_handle> random_directory(const path_handle &dirpath, directory_handle::mode _mode = directory_handle::mode::write, directory_handle::caching _caching = directory_handle::caching::temporary, directory_handle::flag flags = directory_handle::flag::none) noexcept
+{
+  return directory_handle::random_directory(std::forward<decltype(dirpath)>(dirpath), std::forward<decltype(_mode)>(_mode), std::forward<decltype(_caching)>(_caching), std::forward<decltype(flags)>(flags));
+}
+/*! Create a directory handle creating the named directory on some path which
+the OS declares to be suitable for temporary files.
+Note also that an empty name is equivalent to calling
+`random_file(temporary_files_directory())` and the creation
+parameter is ignored.
+
+\errors Any of the values POSIX open() or CreateFile() can return.
+*/
+inline result<directory_handle> temp_directory(directory_handle::path_view_type name = directory_handle::path_view_type(), directory_handle::mode _mode = directory_handle::mode::write, directory_handle::creation _creation = directory_handle::creation::if_needed,
+                                               directory_handle::caching _caching = directory_handle::caching::all, directory_handle::flag flags = directory_handle::flag::none) noexcept
+{
+  return directory_handle::temp_directory(std::forward<decltype(name)>(name), std::forward<decltype(_mode)>(_mode), std::forward<decltype(_creation)>(_creation), std::forward<decltype(_caching)>(_caching), std::forward<decltype(flags)>(flags));
+}
+/*! Fill the buffers type with as many directory entries as will fit.
+
+\return True if the entire directory was read into `tofill`, false otherwise.
+`tofill`'s extent is adjusted to match the number of items read on exit.
+\param self The object whose member function to call.
+\param tofill The buffers to fill.
+\param glob An optional shell glob by which to filter the items filled. Done kernel side on Windows, user side on POSIX.
+\param filtering Whether to filter out fake-deleted files on Windows or not.
+\param kernelbuffer A buffer to use for the kernel to fill. If left defaulted, a kernel buffer
+is allocated internally and stored into `tofill` which needs to not be destructed until one
+is no longer using any items within (leafnames are views onto the original kernel data).
+\errors todo
+\mallocs If the `kernelbuffer` parameter is set on entry, no memory allocations.
+If unset, at least one memory allocation, possibly more is performed.
+*/
+inline result<directory_handle::enumerate_info> enumerate(const directory_handle &self, directory_handle::buffers_type &tofill, directory_handle::path_view_type glob = directory_handle::path_view_type(), directory_handle::filter filtering = directory_handle::filter::fastdeleted,
+                                                          span<char> kernelbuffer = span<char>()) noexcept
+{
+  return self.enumerate(std::forward<decltype(tofill)>(tofill), std::forward<decltype(glob)>(glob), std::forward<decltype(filtering)>(filtering), std::forward<decltype(kernelbuffer)>(kernelbuffer));
+}
 // END make_free_functions.py
 
 AFIO_V2_NAMESPACE_END
diff --git a/include/afio/v2.0/fs_handle.hpp b/include/afio/v2.0/fs_handle.hpp
index bc6d2555..45bfcb01 100644
--- a/include/afio/v2.0/fs_handle.hpp
+++ b/include/afio/v2.0/fs_handle.hpp
@@ -138,8 +138,8 @@ public:
   */
   result<path_handle> parent_path_handle(deadline d = std::chrono::seconds(30)) const noexcept;
 
-  /*! Atomically relinks the current path of this open handle to the new path specified,
-  \b atomically and silently replacing any item at the new path specified. This operation
+  /*! Relinks the current path of this open handle to the new path specified. If `atomic_replace` is
+  true, the relink \b atomically and silently replaces any item at the new path specified. This operation
   is both atomic and silent matching POSIX behaviour even on Microsoft Windows where
   no Win32 API can match POSIX semantics.
 
@@ -152,13 +152,17 @@ public:
 
   \param base Base for any relative path.
   \param newpath The relative or absolute new path to relink to.
+  \param atomic_replace Atomically replace the destination if a file entry already is present there.
+  Choosing false for this will fail if a file entry is already present at the destination, and may
+  not be an atomic operation on some platforms (i.e. both the old and new names may be linked to the
+  same inode for a very short period of time). Windows and recent Linuxes are always atomic.
   \param d The deadline by which the matching of the containing directory to the open handle's inode
   must succeed, else `std::errc::timed_out` will be returned.
   \mallocs Except on platforms with race free syscalls for renaming open handles (Windows), calls
   `current_path()` and thus is both expensive and calls malloc many times.
   */
   AFIO_MAKE_FREE_FUNCTION
-  result<void> relink(const path_handle &base, path_view_type newpath, deadline d = std::chrono::seconds(30)) noexcept;
+  result<void> relink(const path_handle &base, path_view_type newpath, bool atomic_replace = true, deadline d = std::chrono::seconds(30)) noexcept;
 
   /*! Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system.
   On Windows unless `flag::win_disable_unlink_emulation` is set, this behaviour is
@@ -185,8 +189,8 @@ public:
 };
 
 // BEGIN make_free_functions.py
-/*! Atomically relinks the current path of this open handle to the new path specified,
-\b atomically and silently replacing any item at the new path specified. This operation
+/*! Relinks the current path of this open handle to the new path specified. If `atomic_replace` is
+true, the relink \b atomically and silently replaces any item at the new path specified. This operation
 is both atomic and silent matching POSIX behaviour even on Microsoft Windows where
 no Win32 API can match POSIX semantics.
 
@@ -200,13 +204,18 @@ success until the deadline given. This should prevent most unmalicious accidenta
 \param self The object whose member function to call.
 \param base Base for any relative path.
 \param newpath The relative or absolute new path to relink to.
+\param atomic_replace Atomically replace the destination if a file entry already is present there.
+Choosing false for this will fail if a file entry is already present at the destination, and may
+not be an atomic operation on some platforms (i.e. both the old and new names may be linked to the
+same inode for a very short period of time). Windows and recent Linuxes are always atomic.
 \param d The deadline by which the matching of the containing directory to the open handle's inode
-must succeed, else `std::errc::timed_out` will be returned. Not used on platforms with race free
-syscalls for renaming open handles (Windows).
+must succeed, else `std::errc::timed_out` will be returned.
+\mallocs Except on platforms with race free syscalls for renaming open handles (Windows), calls
+`current_path()` and thus is both expensive and calls malloc many times.
 */
-inline result<void> relink(fs_handle &self, const path_handle &base, fs_handle::path_view_type newpath, deadline d = std::chrono::seconds(30)) noexcept
+inline result<void> relink(fs_handle &self, const path_handle &base, fs_handle::path_view_type newpath, bool atomic_replace = true, deadline d = std::chrono::seconds(30)) noexcept
 {
-  return self.relink(std::forward<decltype(base)>(base), std::forward<decltype(newpath)>(newpath), std::forward<decltype(d)>(d));
+  return self.relink(std::forward<decltype(base)>(base), std::forward<decltype(newpath)>(newpath), std::forward<decltype(atomic_replace)>(atomic_replace), std::forward<decltype(d)>(d));
 }
 /*! Unlinks the current path of this open handle, causing its entry to immediately disappear from the filing system.
 On Windows unless `flag::win_disable_unlink_emulation` is set, this behaviour is
@@ -224,8 +233,10 @@ deadline given. This should prevent most unmalicious accidental loss of data.
 
 \param self The object whose member function to call.
 \param d The deadline by which the matching of the containing directory to the open handle's inode
-must succeed, else `std::errc::timed_out` will be returned. Not used on platforms with race free
-syscalls for unlinking open handles (Windows).
+must succeed, else `std::errc::timed_out` will be returned.
+\mallocs Except on platforms with race free syscalls for unlinking open handles (Windows), calls
+`current_path()` and thus is both expensive and calls malloc many times. On Windows, also calls
+`current_path()` if `flag::disable_safety_unlinks` is not set.
 */
 inline result<void> unlink(fs_handle &self, deadline d = std::chrono::seconds(30)) noexcept
 {
diff --git a/include/afio/v2.0/handle.hpp b/include/afio/v2.0/handle.hpp
index 2f4ead79..4b7e7f1b 100644
--- a/include/afio/v2.0/handle.hpp
+++ b/include/afio/v2.0/handle.hpp
@@ -39,11 +39,14 @@ Distributed under the Boost Software License, Version 1.0.
 
 AFIO_V2_NAMESPACE_EXPORT_BEGIN
 
+class fs_handle;
+
 /*! \class handle
 \brief A native_handle_type which is managed by the lifetime of this object instance.
 */
 class AFIO_DECL handle
 {
+  friend class fs_handle;
   friend inline std::ostream &operator<<(std::ostream &s, const handle &v);
 
 public:
@@ -135,8 +138,9 @@ public:
 
     // NOTE: IF UPDATING THIS UPDATE THE std::ostream PRINTER BELOW!!!
 
-    overlapped = 1 << 28,         //!< On Windows, create any new handles with OVERLAPPED semantics
-    byte_lock_insanity = 1 << 29  //!< Using insane POSIX byte range locks
+    overlapped = 1 << 28,          //!< On Windows, create any new handles with OVERLAPPED semantics
+    byte_lock_insanity = 1 << 29,  //!< Using insane POSIX byte range locks
+    anonymous_inode = 1 << 30      //!< This is an inode created with no representation on the filing system
   }
   QUICKCPPLIB_BITFIELD_END(flag);
 
diff --git a/include/afio/v2.0/native_handle_type.hpp b/include/afio/v2.0/native_handle_type.hpp
index b327cb48..49180b39 100644
--- a/include/afio/v2.0/native_handle_type.hpp
+++ b/include/afio/v2.0/native_handle_type.hpp
@@ -84,9 +84,9 @@ struct native_handle_type
     o._init = -1;
   }
   //! Copy assign
-  native_handle_type &operator=(const native_handle_type &) = default;
+  constexpr native_handle_type &operator=(const native_handle_type &) = default;
   //! Move assign
-  native_handle_type &operator=(native_handle_type &&o) noexcept
+  constexpr native_handle_type &operator=(native_handle_type &&o) noexcept
   {
     behaviour = std::move(o.behaviour);
     _init = std::move(o._init);
diff --git a/include/afio/v2.0/path_view.hpp b/include/afio/v2.0/path_view.hpp
index e3cc2f31..a9a22b5f 100644
--- a/include/afio/v2.0/path_view.hpp
+++ b/include/afio/v2.0/path_view.hpp
@@ -161,17 +161,9 @@ private:
   {
     // wchar paths must use backslashes
     if(!_state._utf16.empty())
-      return _state._utf16.rfind(filesystem::path::preferred_separator);
+      return _state._utf16.rfind('\\');
     // char paths can use either
-    auto idx1 = _state._utf8.rfind('\\');
-    auto idx2 = _state._utf8.rfind('/');
-    if(_npos == idx1 && _npos == idx2)
-      return _npos;
-    if(_npos == idx1)
-      return idx2;
-    if(_npos == idx2)
-      return idx1;
-    return (idx1 < idx2) ? idx1 : idx2;
+    return _state._utf8.find_last_of("/\\");
   }
 #else
   struct state
@@ -260,6 +252,19 @@ public:
   constexpr bool has_extension() const noexcept;
   constexpr bool is_absolute() const noexcept;
   constexpr bool is_relative() const noexcept;
+  // True if the path view contains any of the characters `*`, `?`, (POSIX only: `[` or `]`).
+  constexpr bool contains_glob() const noexcept
+  {
+#ifdef _WIN32
+    if(!_state._utf16.empty())
+      return wstring_view::npos != _state._utf16.find_first_of(L"*?");
+    if(!_state._utf8.empty())
+      return wstring_view::npos != _state._utf8.find_first_of("*?");
+    return false;
+#else
+    return string_view::npos != _state._utf8.find_first_of("*?[]");
+#endif
+  }
 #ifdef _WIN32
   // True if the path view is a NT kernel path starting with `\!!\` or `\??\`
   constexpr bool is_ntpath() const noexcept
diff --git a/test/kerneltest b/test/kerneltest
-Subproject 62a1669bae87a47ea4cc05c474e844eb7aca6aa
+Subproject 21bb216d467bc12d4b227a69ff4c2559ccb5a27