THIS IS A TEST INSTANCE ONLY! REPOSITORIES CAN BE DELETED AT ANY TIME!

Browse Source

Add support for different allocation schemes

master
Joel Yliluoma 3 years ago
parent
commit
a9c08050c0
2 changed files with 66 additions and 6 deletions
  1. +19
    -5
      README.md
  2. +47
    -1
      gunzip.hh

+ 19
- 5
README.md View File

@ -40,14 +40,28 @@ To adjust the memory usage, there are three settings in gunzip.hh you can change
| Setting name | 'false' memory use bytes | 'true' memory use bytes | 'true' performance impact
| ------------------------------------------- | ---:| ----:|--------------
| **USE_BITARRAY_TEMPORARY_IN_HUFFMAN_CREATION** | 30 | 24 | Negligible
| **USE_BITARRAY_FOR_LENGTHS** | 288 | 144 | Noticeable
| **USE_BITARRAY_FOR_HUFFNODES** | 653 | 384 | Significant
| `USE_BITARRAY_TEMPORARY_IN_HUFFMAN_CREATION` | 30 | 24 | Negligible
| `USE_BITARRAY_FOR_LENGTHS` | 288 | 144 | Noticeable
| `USE_BITARRAY_FOR_HUFFNODES` | 653 | 384 | Significant
| **Total** | 971 | 552 | _Plus alignment losses, callframes and spills_
You can also change the memory allocation scheme:
| `#define` name | Meaning
| --- | ---
| `DEFLATE_ALLOCATION_AUTOMATIC` | Automatic allocation (usually stack)
| `DEFLATE_ALLOCATION_STATIC` | Static `thread_local` allocation (memory remains allocated throughout the program, and different threads have their own copy of the data). Note that this scheme excludes running multiple decompressions in parallel, unless you do it in different threads.
| `DEFLATE_ALLOCATION_DYNAMIC` | Storage duration is the same as with automatic allocation, but the `new` keyword is explicitly used (which usually means heap/bss allocation).
There is also a constant `MAX_WINDOW_SIZE`, which is 32768 by default,
but you can reduce it to use less memory for the automatically allocated
window in situations where one is allocated (see note 9 below).
Note that this value must not be smaller than the maximum backreference
distance used by your compressed data.
## Unrequirements
* No dynamic memory is allocated under any circumstances, unless your user-supplied functors do it.
* No dynamic memory is allocated under any circumstances, unless your user-supplied functors do it, or you `#define DEFLATE_ALLOCATION_DYNAMIC`.
* Aside from assert() in assert.h and some template metaprogramming tools in type_traits, no standard library functions are used.
* No global variables.
* Compatible with -fno-exceptions -fno-rtti compilation.
@ -64,7 +78,7 @@ To adjust the memory usage, there are three settings in gunzip.hh you can change
* Slower than your average inflate function. The template uses densely bitpacked arrays, which require plenty of bit-shifting operations for every access.
* The code obviously performs best on 32-bit or 64-bit platforms. Platforms where 32-bit entities must be synthesized from a number of 8-bit entities are at a disadvantage.
* Decompressed data integrity is not verified. Any checksum fields are totally ignored.
* On most systems, automatic storage means ‘stack allocation’. Depending on your circumstances, you may want to add a `static` (and maybe `thread_local`) into `gunzip_ns::DeflateState state;` to use static allocation instead. Or, use `std::unique_ptr` to change it into dynamic allocation.
* On most systems, automatic storage means ‘stack allocation’. Depending on your circumstances, you may want to change the memory allocation scheme. See the Tuning chapter for details.
## Definitions

+ 47
- 1
gunzip.hh View File

@ -86,7 +86,15 @@ namespace gunzip_ns
static_assert(MAX_WINDOW_SIZE >= 2, "Max window size should be >= 2");
static_assert(MAX_WINDOW_SIZE <= 32768u, "Window sizes larger than 32768 are not supported by deflate standard. Edit the source code to remove this assert if you need it.");
//
#define DEFLATE_USE_DATA_TABLES
#if !defined(DEFLATE_ALLOCATION_AUTOMATIC) && !defined(DEFLATE_ALLOCATION_STATIC) && !defined(DEFLATE_ALLOCATION_DYNAMIC)
// Choose one:
#define DEFLATE_ALLOCATION_AUTOMATIC
//#define DEFLATE_ALLOCATION_STATIC
//#define DEFLATE_ALLOCATION_DYNAMIC
#endif
}
#define DeflIsInputFunctor (std::is_convertible<std::result_of_t<std::decay_t<InputFunctor>()>,int>::value /*\
@ -131,6 +139,10 @@ namespace gunzip_ns
#define DeflReturnType std::result_of_t<gunzip_ns::SizeTracker<SizeTrackTag>(int)>
#ifdef DEFLATE_ALLOCATION_DYNAMIC
# include <memory>
#endif
// RandomAccessBitArray: An engine for arrays of data items that are not byte-aligned
template<typename U = std::uint_least64_t>
struct RandomAccessBitArrayBase
@ -636,6 +648,23 @@ namespace gunzip_ns
typedef std::pair<int, std::pair<std::uint_fast64_t,std::uint_fast64_t>> result;
inline result operator() (int returncode) const { return result{returncode,std::make_pair(insize,outsize)}; }
};
#ifdef DEFLATE_ALLOCATION_STATIC
template<bool=0>
gunzip_ns::DeflateState& GetState()
{
static thread_local gunzip_ns::DeflateState state;
state.~DeflateState();
new(&state) DeflateState();
return state;
}
template<bool=0>
gunzip_ns::DeflateWindow& GetWindow()
{
static thread_local gunzip_ns::DeflateWindow window;
return window;
}
#endif
}//ns
/* Values of Abortable:
@ -815,7 +844,15 @@ template
std::enable_if_t<DeflIsInputFunctor && DeflIsOutputFunctor && DeflIsWindowFunctor && DeflIsTrackTag, DeflReturnType>
Deflate(InputFunctor&& input, OutputFunctor&& output, WindowFunctor&& outputcopy, SizeTrackTag = {})
{
#ifdef DEFLATE_ALLOCATION_AUTOMATIC
gunzip_ns::DeflateState state;
#elif defined(DEFLATE_ALLOCATION_STATIC)
auto& state = gunzip_ns::GetState<>();
#elif defined(DEFLATE_ALLOCATION_DYNAMIC)
std::unique_ptr<gunzip_ns::DeflateState> stateptr(new gunzip_ns::DeflateState);
auto& state = *stateptr;
#endif
gunzip_ns::SizeTracker<SizeTrackTag> tracker;
// For output to be abortable, both the output functor and window functor must have return types.
@ -833,9 +870,18 @@ std::enable_if_t
Deflate(InputFunctor&& input, OutputFunctor&& output, SizeTrackTag = {})
{
// Using a library-supplied output window. OutputFunctor will be wrapped.
#ifdef DEFLATE_ALLOCATION_AUTOMATIC
gunzip_ns::DeflateState state;
gunzip_ns::DeflateWindow window;
#elif defined(DEFLATE_ALLOCATION_STATIC)
auto& state = gunzip_ns::GetState<>();
auto& window = gunzip_ns::GetWindow<>();
#elif defined(DEFLATE_ALLOCATION_DYNAMIC)
std::unique_ptr<std::pair<gunzip_ns::DeflateState, gunzip_ns::DeflateWindow>> dataptr
(new std::pair<gunzip_ns::DeflateState, gunzip_ns::DeflateWindow>);
auto& state = dataptr->first;
auto& window = dataptr->second;
#endif
gunzip_ns::SizeTracker<SizeTrackTag> tracker;
constexpr unsigned char Abortable = DeflInputAbortable_InputFunctor

Loading…
Cancel
Save