基于Facebook开源的 rocksdb 分布式数据库模块 [迭代]

Buei

由于之前帖子内容太多,这里重新发布一篇,之前帖子地址--> 地址


什么是rocksdb?

• RocksDB是一个 持久的键值存储库 ，它是用C++编写的，适合在快速、低延迟的存储设备上存储数据。它是由Facebook数据库工程团队开发和维护。
  

LevelDB 有什么区别?

• RocksDB和LevelDB都是基于LSM-Tree的嵌入式键值存储库，但RocksDB是在LevelDB的基础上进行了优化和增强
  
  
  

• RocksDB可以支持 多线程 合并文件，而LevelDB是 单线程
• RocksDB可以根据需要开辟 多个Memtable ，而LevelDB只有 一个Memtable
• RocksDB可支持多种压缩算法，而LevelDB只支持snappy
  
• 单线程模式下 LevelDB 可能稍微快一点,而在多线程下 RocksDB 就会发挥出它的优势了
  
  

rocksdb的优点

• 高性能：RocksDB 使用了很多优化技术，如多线程、高效的数据结构等，因此具有非常高的读写性能。
• 可扩展性：RocksDB 可以处理大规模的数据，并支持自动分片和负载均衡等功能，因此可以很好地应对高并发访问。
• 可靠性：RocksDB 支持 ACID 事务，保证数据的一致性和可靠性。
• 灵活性：RocksDB 支持多种数据格式，包括内存映射文件、纯内存等，让用户可以灵活选择适合自己的存储方式。
• RocksDB在存储数据时是按照键的排序方式进行存储的，它并没有明确的容量限制，可以存储非常大的数据 [理论上无限制容量]。而类似MMKV框架限制容量的方式是使用了一种固定大小的映射文件，即在创建MMKV实例时就已经确定了最大容量，超过容量时就不能再写入数据[大概在 4GB 左右]
  

• 另一款开源 leveldb的帖子
  

下图为 rocksdb和leveldb  单/多线程 写入对比
理论上在单线程下 RocksDB应该比levldb稍微略慢一点

图中可看到300W多线程写入RocksDB一瞬间完成

解压密码:neko

另外发布了一份火山版rocksdb

更新日志  2023/09/21 18:00  - V1.5  本次更新内容有点多

• 本次使用了C++ MT 模式编译
• 增加 Transaction事务类   (此事务支持在事务内创建迭代器,支持事务读取回滚操作)
• 增加 OptimisticTransactionDB_类
• 增加 TransactionDB类 (TransactionDB(PessimisticTransactionDB)和OptimisticTransactionDB，分别对应并发控制中的悲观锁和乐观锁)
• 增加 Merge 合并功能   (使用此命令前务必在 rocksdb_启动参数 的 合并模式 选择合并模式)
• 修复 压缩模式 无效
  

• 修复 关闭写前日志 无效    (如果想完全启用内存模式,请将缓存调大关闭写前日志开启 即可)
  

• OptimisticTransactionDB，TransactionDB  对应的事务 [Transaction事务类]
• ...
  

• 以下为Facebook官方更新
• -Please note 8.5.1 includes a fix for a persisted database corruption in an unlikely edge case. Upgrading to a version including this fix, like this one, is highly recommended!
  

• Fixed a race condition in GenericRateLimiter that could cause it to stop granting requests
• Fix a bug where iterator may return incorrect result for DeleteRange() users if there was an error reading from a file.
• Fix a bug where if there is an error reading from offset 0 of a file from L1+ and that the file is not the first file in the sorted run, data can be lost in compaction and read/scan can return incorrect results.
  

• https://wwqr.lanzouw.com/b09ivrlpc
• 密码:7l4b
  

Buei

问题反馈/交流Q群

Q群

Buei

更新日志  2023/11/28 15:00  - V1.6 [rocksdb内核版本:8.8.1]

增加 设置  预读取缓存大小 [默认未开启
增加 设置  可见性等级  
增加 设置  日志备份次数 [LOG.old 开头的文件每次启动数据库时备份次数]
增加 自定义日志目录
修复 当value为空时写入失败
...



以下为Facebook官方更新日志

8.8.1 (2023-11-17)Bug fixes

• Make the cache memory reservation accounting in Tiered cache (primary and compressed secondary cache) more accurate to avoid over/under charging the secondary cache.
• Allow increasing the compressed_secondary_ratio in the Tiered cache after setting it to 0 to disable.
  

8.8.0 (2023-10-23)New Features

• Introduce AttributeGroup by adding the first AttributeGroup support API, MultiGetEntity(). Through the use of Column Families, AttributeGroup enables users to logically group wide-column entities. More APIs to support AttributeGroup will come soon, including GetEntity, PutEntity, and others.
• Added new tickers rocksdb.fifo.{max.size|ttl}.compactions to count FIFO compactions that drop files for different reasons
• Add an experimental offpeak duration awareness by setting DBOptions::daily_offpeak_time_utc in "HH:mm-HH:mm" format. This information will be used for resource optimization in the future
• Users can now change the max bytes granted in a single refill period (i.e, burst) during runtime by SetSingleBurstBytes() for RocksDB rate limiter
  

Public API Changes

• The default value of DBOptions::fail_if_options_file_error changed from false to true. Operations that set in-memory options (e.g., DB::Open*(), DB::SetOptions(), DB::CreateColumnFamily*(), and DB::DropColumnFamily()) but fail to persist the change will now return a non-OK Status by default.
• Add new Cache APIs GetSecondaryCacheCapacity() and GetSecondaryCachePinnedUsage() to return the configured capacity, and cache reservation charged to the secondary cache.
  

Behavior Changes

• For non direct IO, eliminate the file system prefetching attempt for compaction read when Options::compaction_readahead_size is 0
• During a write stop, writes now block on in-progress recovery attempts
• Deleting stale files upon recovery are delegated to SstFileManger if available so they can be rate limited.
  

Bug Fixes

• Fix a bug in auto_readahead_size where first_internal_key of index blocks wasn't copied properly resulting in corruption error when first_internal_key was used for comparison.
• Fixed a bug where compaction read under non direct IO still falls back to RocksDB internal prefetching after file system's prefetching returns non-OK status other than Status::NotSupported()
• Add bounds check in WBWIIteratorImpl and make BaseDeltaIterator, WriteUnpreparedTxn and WritePreparedTxn respect the upper bound and lower bound in ReadOption. See 11680.
• Fixed the handling of wide-column base values in the max_successive_merges logic.
• Fixed a rare race bug involving a concurrent combination of Create/DropColumnFamily and/or Set(DB)Options that could lead to inconsistency between (a) the DB's reported options state, (b) the DB options in effect, and (c) the latest persisted OPTIONS file.
• Fixed a possible underflow when computing the compressed secondary cache share of memory reservations while updating the compressed secondary to total block cache ratio.
  

Performance Improvements

• Improved the I/O efficiency of DB::Open a new DB with create_missing_column_families=true and many column families.
  

• https://wwqr.lanzouw.com/b09ivrlpc
• 密码:7l4b
  

JYYeah

感谢分享，很给力！~

ts0916

感谢分享，很给力！~

ctry78985

感谢分享

Adys

可怕强的！！！！！！！！！！！！！！！！！！

人头收割机

感谢分享，很给力！~

mmlai8

牛皮，支持,谢谢分享。

shaokui123

感谢分享

wmv520

开源精神必须支持~        YYDS~！

五花牛

感谢楼主分享！

shaokui123

感谢分享