1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
|
Herein lies an exploratory toy ACID key-value store written using
AFIO which lets you look up any BLOB value from some 128-bit key, and to
update as an atomic transaction up to 65,535 key-values at once.
It is purely to test the feasibility of one approach to implementing such
a store, and to test AFIO's design. Nobody should use this store for
anything serious.
## Todo:
- [x] Add sparse file creation on Windows to AFIO and see how the
benchmarks fare.
- [x] Add key-value deletion.
- [x] Atomic append should issue gather buffers of `IOV_MAX`
- [x] Optionally use mmaps to extend smallfile instead of atomic appends.
Likely highly racy on Linux due to kernel bugs :)
- [x] Use mmaps for all smallfiles
- [ ] Does this toy store actually work with multiple concurrent users?
- [ ] Online free space consolidation (copy early still in use records
into new small file, update index to use new small file)
- [ ] Per 1Mb free space consolidated, punch hole
- [ ] Need some way of detecting and breaking sudden process exit during
index update.
## Benchmarks:
- 1Kb values Windows with NTFS, no integrity, no durability, read + append:
```
Inserting 1M key-value pairs ...
Inserted at 195312 items per sec
Retrieving 1M key-value pairs ...
Fetched at 612745 items per sec
```
- 1Kb values Windows with NTFS, integrity, no durability, read + append:
```
Inserting 1M key-value pairs ...
Inserted at 188572 items per sec
Retrieving 1M key-value pairs ...
Fetched at 542005 items per sec
```
- 1Kb values Windows with NTFS, no integrity, no durability, mmaps:
```
Inserting 1M key-value pairs ...
Inserted at 518403 items per sec
Retrieving 1M key-value pairs ...
Fetched at 2192982 items per sec
```
- 1Kb values Windows with NTFS, integrity, no durability, mmaps:
```
Inserting 1M key-value pairs ...
Inserted at 455996 items per sec
Retrieving 1M key-value pairs ...
Fetched at 1144164 items per sec
```
- 1Kb values Linux with ext4, no integrity, no durability:
```
Inserting 1M key-value pairs ...
Inserted at 656598 items per sec
Retrieving 1M key-value pairs ...
Fetched at 1945525 items per sec
```
- 1Kb values Linux with ext4, integrity, no durability:
```
Inserting 1M key-value pairs ...
Inserted at 581057 items per sec
Retrieving 1M key-value pairs ...
Fetched at 1519756 items per sec
```
- 16 byte values Windows with NTFS, no integrity, no durability, read + append:
```
Inserting 1M key-value pairs ...
Inserted at 214178 items per sec
Retrieving 1M key-value pairs ...
Fetched at 660066 items per sec
```
- 16 byte values Windows with NTFS, no integrity, no durability, mmaps:
```
Inserting 1M key-value pairs ...
Inserted at 938967 items per sec
Retrieving 1M key-value pairs ...
Fetched at 4739336 items per sec
```
- 16 byte values Linux with ext4, no integrity, no durability, read + append:
```
Inserting 1M key-value pairs ...
Inserted at 1118568 items per sec
Retrieving 1M key-value pairs ...
Fetched at 2898550 items per sec
```
|
