From fce40169bd7e0cb457c54f89c2a3f5c70fd03278 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=9B=D0=B5=D0=BE=D0=BD=D0=B8=D0=B4=20=D0=AE=D1=80=D1=8C?= =?UTF-8?q?=D0=B5=D0=B2=20=28Leonid=20Yuriev=29?= Date: Sun, 19 Jan 2025 02:14:19 +0300 Subject: [PATCH] =?UTF-8?q?mdbx-doc:=20=D0=B4=D0=BE=D1=80=D0=B0=D0=B1?= =?UTF-8?q?=D0=BE=D1=82=D0=BA=D0=B0/=D0=B0=D0=BA=D1=82=D1=83=D0=B0=D0=BB?= =?UTF-8?q?=D0=B8=D0=B7=D0=B0=D1=86=D0=B8=D1=8F=20=D1=80=D0=B0=D0=B7=D0=B4?= =?UTF-8?q?=D0=B5=D0=BB=D0=B0=20"Restrictions=20&=20Caveats".?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/_restrictions.md | 79 +++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 30 deletions(-) diff --git a/docs/_restrictions.md b/docs/_restrictions.md index 7e905b5e..eb488a43 100644 --- a/docs/_restrictions.md +++ b/docs/_restrictions.md @@ -42,18 +42,33 @@ long-lived read transactions, and using the \ref MDBX_LIFORECLAIM mode which addresses subsequent performance degradation. The "next" version of libmdbx (aka \ref MithrilDB) will completely solve this. -- Avoid suspending a process with active transactions. These would then be - "long-lived" as above. +Nonetheless, situations that encourage lengthy read transactions while +intensively updating data should be avoided. For example, you should +avoid suspending/blocking processes/threads performing read +transactions, including during debugging, and use transaction parking if +necessary. -- Avoid aborting a process with an active read-only transaction in scenarios - with high rate of write transactions. The transaction becomes "long-lived" - as above until a check for stale readers is performed or the LCK-file is - reset, since the process may not remove it from the lockfile. This does - not apply to write transactions if the system clears stale writers, see - above. +You should also beware of aborting processes that perform reading +transactions. Despite the fact that libmdbx automatically checks and +cleans readers, as an a process aborting (especially with core dump) can +take a long time, and checking readers cannot be performed too often due +to performance degradation. + +This issue will be addressed in MithrlDB and one of libmdbx releases, +presumably in 2025. To do this, nonlinear GC recycling will be +implemented, without stopping garbage recycling on the old MVCC snapshot +used by a long read transaction. + +After the planned implementation, any long-term reading transaction will +still keep the used MVCC-snapshot (all the database pages forming it) +from being recycled, but it will allow all unused MVCC snapshots to be +recycled, both before and after the readable one. This will eliminate +one of the main architectural flaws inherited from LMDB and caused the +growth of a database in proportion to a volume of data changes made +concurrently with a long-running read transaction. -## Large data items and huge transactions +## Large data items MDBX allows you to store values up to 1 gigabyte in size, but this is not the main functionality for a key-value storage, but an additional @@ -71,6 +86,18 @@ and in the absence of a sufficient sequence of free pages, increase the DB file. Thus, for long values, MDBX provides maximum read performance at the expense of write performance. +Some aspects related to GC have been refined and improved in 2022 within +the first releases of the 0.12.x series. In particular the search for +free consecutive/adjacent pages through GC has been significantly +speeded, including acceleration using NOEN/SSE2/AVX2/AVX512 +instructions. + +This issue will be addressed in MithrlDB and refined within one of +0.15.x libmdbx releases, presumably at end of 2025. + + +### Huge transactions + A similar situation can be with huge transactions, in which a lot of database pages are retired. The retired pages should be put into GC as a list of page numbers for future reuse. But in huge transactions, such a @@ -80,20 +107,15 @@ you delete large amounts of information from the database in a single transaction, MDBX may need to increase the database file to save the list of pages to be retired. -Both of these issues will be addressed in MithrilDB. +This issue was fixed in 2022 within the first releases of the 0.12.x +series by `Big Foot` feature, which now is enabled by default. +See \ref MDBX_ENABLE_BIGFOOT build-time option. -#### Since v0.12.1 and later -Some aspects related to GC have been refined and improved, in particular: - - - The search for free consecutive/adjacent pages through GC has been - significantly speeded, including acceleration using - NOEN/SSE2/AVX2/AVX512 instructions. - - - The `Big Foot` feature which significantly reduces GC overhead for - processing large lists of retired pages from huge transactions. Now - libmdbx avoid creating large chunks of PNLs (page number lists) which - required a long sequences of free pages, aka large/overflow pages. Thus - avoiding searching, allocating and storing such sequences inside GC. +The `Big Foot` feature which significantly reduces GC overhead for +processing large lists of retired pages from huge transactions. Now +libmdbx avoid creating large chunks of PNLs (page number lists) which +required a long sequences of free pages, aka large/overflow pages. Thus +avoiding searching, allocating and storing such sequences inside GC. ## Space reservation @@ -135,9 +157,11 @@ On the other hand, MDBX allow calling \ref mdbx_env_close() in such cases to release resources, but no more and in general this is a wrong way. #### Since v0.13.1 and later -Начиная с версии 0.13.1 в API доступна функция \ref mdbx_env_resurrect_after_fork(), -которая позволяет пере-использовать в дочерних процессах уже открытую среду БД, -но строго без наследования транзакций от родительского процесса. + +Starting from the v0.13.1 release, the \ref mdbx_env_resurrect_after_work() +is available, which allows you to reuse an already open database +environment in child processes, but strictly without inheriting any +transactions from a parent process. ## Read-only mode @@ -149,9 +173,6 @@ to without-LCK mode on appropriate errors (`EROFS`, `EACCESS`, `EPERM`) if the read-only mode was requested by the \ref MDBX_RDONLY flag which is described below. -The "next" version of libmdbx (\ref MithrilDB) will solve this issue for the "many -readers without writer" case. - ## Troubleshooting the LCK-file 1. A broken LCK-file can cause sync issues, including appearance of @@ -166,8 +187,6 @@ readers without writer" case. multiple processes in a lock-free manner and any locking is unwise due to a large overhead. - The "next" version of libmdbx (\ref MithrilDB) will solve this issue. - \note Workaround: Just make all programs using the database close it; the LCK-file is always reset on first open.