0% found this document useful (0 votes)
10 views7 pages

6.database-Oriented Sotrage Based On LMDB and Linear Octree For Massive Block Model

Uploaded by

Sebastian Avalos
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
10 views7 pages

6.database-Oriented Sotrage Based On LMDB and Linear Octree For Massive Block Model

Uploaded by

Sebastian Avalos
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Trans. Nonferrous Met. Soc.

China 26(2016) 2462−2468

Database-oriented storage based on LMDB and


linear octree for massive block model

Lin BI, Hui ZHAO, Ming-tao JIA


School of Resources and Safety Engineering, Central South University, Changsha 410083, China
Received 14 January 2016; accepted 12 June 2016

Abstract: Data organization requires high efficiency for large amount of data applied in the digital mine system. A new method of
storing massive data of block model is proposed to meet the characteristics of the database, including ACID-compliant, concurrency
support, data sharing, and efficient access. Each block model is organized by linear octree, stored in LMDB (lightning
memory-mapped database). Geological attribute can be queried at any point of 3D space by comparison algorithm of location code
and conversion algorithm from address code of geometry space to location code of storage. The performance and robustness of
querying geological attribute at 3D spatial region are enhanced greatly by the transformation from 3D to 2D and the method of 2D
grid scanning to screen the inner and outer points. Experimental results showed that this method can access the massive data of block
model, meeting the database characteristics. The method with LMDB is at least 3 times faster than that with etree, especially when it
is used to read. In addition, the larger the amount of data is processed, the more efficient the method would be.
Key words: block model; linear octree; lightning memory-mapped database; mass data access; digital mine; etree

meets the characteristics of the database: ACID-


1 Introduction compliant and concurrency support; 2) The data access
should be fast enough, especially the attribute query
3D geological modeling is composed of geological based on any spatial point and any spatial 3D area. In
structure modeling and attribute modeling [1], which is this work, thanks to the linear octree and LMDB, the
the basis for digital mine as well as intelligent mine [2]. efficient storage of massive data of block model was
The structure model, namely surface model, can well realized to meet the characteristics of the database by
describe the space shape of complex ore body, but the making comparative analysis with the implementation
attributes of any point inside the model need to be based on etree. Meanwhile, the screening algorithm, by
expressed and carried by attribute model because of means of 2D grid scanning, improved the efficiency and
inhomogeneity of geological grade. The attribute robustness of the regional query greatly.
modeling of geological body usually means discreting
the geological body into a mesh of voxels [3] which can 2 Expression of block model
be tetrahedral, hexahedral, prism, polyhedral, etc. The
attribute model composed of hexahedral voxels is 2.1 Way of expression
generally called block model. The majority mining Based on the geological structure model, the block
softwares (Vulcan, Data mine, Surpac, Mine Sight, model focused on 3D rasterization. As shown in Fig. 1
Micro mine, Dimine, QuantyMine, 3DMine, etc.) use (using 2D to express for convenience), further
block model as the attribute model. The data quantity of subdivision of the boundary of the ore body is required
block model is huge, even massive. In the application of to meet the accuracy requirement. The block model was
mining software, the following requirements are usually expressed with octree. As a result, complex
necessary for the block model: 1) Supporting geological body with complex boundary can be
concurrency and sharing for massive data of geological accurately interpreted [4,5]. The octree can not only
attributes, in which it requires that the storage method represent accurately the features of complex geological

Foundation item: Projects (41572317, 51374242) supported by the National Natural Science Foundation of China; Project (2015CX005) supported by the
Innovation Driven Plan of Central South University, China
Corresponding author: Lin BI; Tel: +86-731-88877665; E-mail: [email protected]
DOI: 10.1016/S1003-6326(16)64377-7
Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468 2463
The size of the node at k level is 2(31−k) basic units. A
block model with 13 bytes as the location code is
sufficient to express any scale of mine and meets the
accuracy requirements. For example, for a mine of
1000 km, the basic unit length can be 0.5 mm.

Fig. 1 Discrete geological bodies

Fig. 2 Octree area descriptions


body boundary, but also take a great significance in
which the storage space could become smaller, generally
only 10%−30% of 3D grid data [6]. In addition, the
3 Storage of block model
hierarchical structure of octree can increase the speed of
As mentioned above, the block model is represented
retrieval, subdivision, merging and so on.
by linear octree. The linear octree is a technique for
Owing to the lack in regularity as the grid model,
the octree model does not act as well as the grid model in assigning unique addresses to octants [8,9]. It allows
querying. In consideration of the requirements of octants to be located quickly, without actually storing the
massive data, this paper adopts the encoding of linear topological structure of the octree on disk. The linear
octree in which address code is usually represented by octree only save and deal with leaf nodes, and the leaf
Morton code [7]. The Morton code maps a point on the node consists of two parts: the location code and attribute
d-dimensional space to an integer, which can be realized data. This feature is suitable for the key-value database
by exchanging the bits of Cartesian coordinate. to store data. The key-value database is one of the
important types of NoSQL [10] database. NoSQL
2.2 Location code database is to solve the challenges of large-scale data
Figure 2 shows that a domain is a Cartesian collection of multiple data types, especially for the big
coordinate space that consists of a uniform grid of 2n×2n data application problems. Both etree [11] and
indivisible units. The level 0 at the root covers the entire LMDB [12] are key-value storage databases. Their basic
domain, and each sub-node ratio increases 1 level than idea is to represent all input and output datasets stored on
its parent. The address of each block is determined by the disk, so as to support the massive data storage. Based on
Cartesian coordinate (x, y, z) of its lower left corner and these two different techniques, we designed two storage
its level. For example, b is (0, 0, 0, 1), c is (8, 0, 0, 1), solutions for block model, and made some comparative
and j is (6, 10, 0, 3). analysis, then found the most appropriate solution.
The higher the level of the octree is, the more
precise the block model can be subdivided; thus larger 3.1 Storage based on etree
storage space will be required. In order to facilitate The etree is a database-oriented method for large
realization of location code comparison function, out-of-core octree mesh generation. It is widely used in
considering efficiency, accuracy and other factors, large data access [13−15]. The main idea is to map an
location codes are represented by 13 bytes. Each octree to a database structure and perform all octree
dimension of Cartesian coordinate is represented by operations by querying and updating the database. The
4 bytes, 12 bytes for the three-dimensional, 1 byte for the etree only achieves the storage of massive data, but does
node type (leaf or internal node) and level. Obviously, not meet many features of the database, such as ACID,
the location of 13 bytes can define up to 32 levels (0−31) transaction and concurrency.
of the octree. The node at 0 level is the largest one Figure 3 shows the structure of etree, and it is based
covering the entire address space, while the smallest on B+ tree. B+ tree has been used in database systems for
node is at the 31st level, whose size is one basic unit. decades, and is well-suited for efficient data storage and
2464 Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468
retrieval [16]. Etree stores a series of fixed length records to access on-disk data, leaving caching and buffer-
that contain location codes and sorted by location code. management up to the file system and OS instead of
Each record, indexed by B+ tree, consists of two parts: LMDB itself. For this approach to be viable, the memory
location code and attributes. address space of the operating system that LMDB is
running on must be large enough to contain the expected
data volume. For 32 bit systems, this sets a hard limit on
4 GB. In recent years, 64 bit systems have became more
common, which ups the limit to between hundreds of
terabytes and multiple exabytes [17].

Fig. 3 Etree structure of block model

There are two types of nodes in an etree: the leaf


nodes and the index nodes. The leaf nodes contain data
to be searched. The structure of a leaf node is an array of
records with the form <key, data>, shown in Fig. 4(a). Fig. 5 Copy-on-write B+ tree operations
The index nodes contain routing information to guide the
search for a given key value. The structure of an index Based on the above technology, the LMDB has the
node is an array of pairs <key, pointer>, as shown in following major features: Key/value store implemented
Fig. 4(b). These nodes are mapped to disk pages, so the using B+ trees: ordered-map interface with sorted keys;
pointers are actually disk page number. support for range lookups; fully transactional;
The block model data are stored in the disk file by concurrency support, and so on.
B+ tree index, sorted by location code that can solve the More advanced features include support for storage
bottleneck problem of massive data management and fast of multiple databases in a single file, nested transactions,
access. and storage of multiple values for a single key. So
LMDB is very suitable for the storage of massive block
model, and meets the features of database.
Unlike etree’s node, LMDB has only one node type
illustrated in Fig. 6. First, two unsigned shorts are stored,
which together denote the size of the key-value pair’s
value (for leaf-pages) or the page-number the node
points to (for branch-page). Next is a field used to store
Fig. 4 B+tree node: (a) Leaf node; (b) Index node additional information about the node, then the length of
the key-value pair’s key. Finally, there comes a variable-
3.2 Storage based on LMDB length array used to store the actual key- and value-data.
Lightning memory-mapped database (LMDB) is an
embedded key-value data store developed by Symas,
written in C [12,17]. It was developed for the OpenLDA Fig. 6 Format of single LMDB node
Project with the aim of replacing Berkeley DB
(BDB [18]) as its storage backend, and improved from By using this kind of structure, LMDB is able to
the following aspects: quickly locate where a key-value pair is stored in a page,
1) LMDB stores data in a copy-on-write B+ tree. as well as to add a new key-value pair without having to
Figure 5 illustrated this approach. By implementing a move around a large amount of data.
copy-on-write variant of B+ trees, LMDB is able to
provide a form of multi-version concurrency control 4 Query and retrieval
(MVCC), allowing write- and read-transactions to
operate concurrently. 4.1 Comparison of location code
2) One of the initial goals of the LMDB project is to Location code is designed to establish the index and
create a system where no buffer and cache configuration to access the node data quickly and accurately, so the
and management are necessary. LMDB achieves this by location code comparison algorithm is very important.
using the operation system’s memory-mapping capability For the purpose of comparison, we treat the byte
Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468 2465
sequence as a large positive integer. The length (13-byte) 4.2 Attribute query for any point in space
of the location code byte sequence may be longer than The location code corresponding to the point must
any integer data type. To enable the comparison between be calculated from the Cartesian coordinate when
two conceptually large integers, we perform a integer- querying the attribute at one point of space. Assuming
wise comparison, starting from the most significant that the Cartesian coordinate of the point is (x, y, z), the
integer of the sequence down to the least significant coordinate of lower left corner of the block model is
integer, as shown in Fig. 7(a), integer pair <I1,I2> (xorigin, yorigin, zorigin), and the range of the block model is
compared first, then <J1,J2> compared, then <K1,K2> (xlen, ylen, zlen), the address (ax, ay, az) of the point can be
compared. The comparison function returns the calculated from the following formula:
relationship between two levels from the 13th byte if the
three integers are equal. Using this comparison function ax   x  xorigin  / ( xlen /0  80000000)
  
to sort the octants of an octree in increasing locational 
a y   y  yorigin  / ( ylen /0  80000000) (1)
code order, the ordering of octants is exactly the same as 
that processed by the preorder traversal of the octree. By az   z  zorigin  / ( zlen /0  80000000)
preorder traversal, we mean visiting the root, and then
recursively visiting its children in directional code order, As mentioned above, location code is composed of
as shown in Fig. 7(b). Figure 7(c) shows nodes in address code and level, but it is difficult to get level of
ascending order by location code. points. Luckily, according to the characteristics 2) of
Given these observations, we can get the location code comparison algorithm, applications can
characteristics of location code comparison algorithm: query arbitrary points without any knowledge of the 4
1) The location code of root is the smallest; 2) The structure of the octree. That is why we can use the
location code of subtree’s root is less than location code maximum level (31) as the level to compose the location
of any child nodes; 3) If the location node of A is less code.
than location node of B, the location code of any sub
node of A is less than location code of any sub node of B. 4.3 Atribute query for arbitrary 3D region
Given these features, we can use the locational code of In practical application, geological attribute within
the query unit as a search key and ask the database to 3D region (polyhedral) often wants to be queried, for
return the octant whose locational code is the maximum example, the reserves of ore body, the average grade of
among all the locational codes that are less than the stope, the metal amount of stope, the dilution and loss of
search key. Thus, we are guaranteed to locate the leaf mining, the byproduct in tunneling, optimization in
octant only that the query pixel is enclosed in the leaf open-pit [19], and so on. These problems can be
octant. attributed to the problem of “query for polyhedral from
Both etree and LMDB can customize key- block model”. Luckily, the problem can be translated
comparator function determining ordering among into “query for point” problem by polyhedron
different keys. Etree has a handler to application rasterization that polyhedrons are discretized into voxels
comparison function (etree_compar_t *compar), and and query by the centers of voxels.
LMDB has an API (mdb_set_compar) to set the client- As mentioned above, in condition that the correct
specified key-comparator function. attribute can be hit at any point, the efficiency and

Fig. 7 Preorder traversal of octree: (a) Comparison algorithm diagram; (b) Preorder traversal; (c) Location code
2466 Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468
robustness of the polyhedron rasterization are important
factors. The important technology of discretization is the 5 Experiments and analysis
fast and accurate judgment of the relation between the
points and the polyhedron. BI et al [4] optimized the ray The storage method was realized by VC++, running
method to judge the internal and external relations with on windows7 64 bits OS, CPU was Intel (R) Core (TM)
the OBB tree and node intersection test, and the i5-4750 @ 3.20 GHZ CPU, 4G memory, hard drives are
performance has improved, but the robustness is poor. HDD, 7200 RPM. The scope of one mining was
JING et al [20] improve the robustness with Feito− (3450.0 m, 3862.0 m, 812.0 m), its structure model was
Torres method, but performance has declined. LI shown in Fig. 8(a) and its block model was shown in
et al [21] used flood-fill to to reduce the time complexity Fig. 8(c). The query region was shown in Fig. 8(b), and
to O(n); however, a large number of triangle intersection its size was (1361.0 m, 1462.0 m, 715.0 m), and its block
operations need to be carried out. For solving those model was shown in Fig. 8(d). A comparative analysis
problems, the decision problem of point in polyhedron is was carried out between LMDB and etree according to
transformed into a point in polygon in which the 3D different three scenarios: the same fields but different
problem is transformed into 2D problems. As a result, number of blocks, the same number of blocks but
the problem is greatly simplified. The ray method is a different fields and different key-comparator functions.
common method to judge the relationship between the
5.1 Experiment 1: Same fields but different numbers
point and polygon [22]. Based on the idea of ray method,
of blocks
we propose a new method, 2D grid scanning to screen
The fields remained unchanged, 3 doubles and one
the inner or outer points. The process is as follows:
string of 32 bytes, as well as the number of blocks was
1) Rasterize the target of 3D space based on the unit
increased by 8 times (the level of octree were
size of block model and create 3D grid;
incremented by 1). The two storage methods was
2) Cut the polyhedron with plane through the center
compared by analyzing the data size, consuming time to
of each layer and create 2D contour lines;
create, consuming time to query, and so on. The results
3) Screen the inner or outer of pixels for every layer
were shown in Table 1.
by the raster scan method.
As can be seen, LMDB occupies more storage space
Among them, the raster scanning method is very
than etree. The reason is that LMDB can save
important. Assuming the row, column, layer of the 3D
variable-length records, and needs to hold the length of
grid is N (N=2i, i is a non-zero positive integer), its
key and value. Nonetheless, the feature of variable-
process is as follows:
length key and variable-length value is lamb’s advantage
1) Initialize all elements “outside” mark;
that can be used to store data collection of multiple data
2) Produce a ray Rxj from x=xj parallel to the y-axis; types. LMDB is much faster than etree, and the more the
xj is x-component of coordinate of column j, number of blocks is, the more obvious the tendency is.
3) Compare xj and x-component (x1, x2) of all of the The main reason is that the entire database of LMDB is
end points of contour lines on k layer. There is an mapped into virtual memory and all data fetched are
intersection between line segment and Rxj, if x1<xj<x2 performed via direct access to the mapped memory
(without loss of generality, assuming x1<x2); instead of through intermediate buffers and copies. In
4) Calculate the y-component value of the addition, the screening is time-used less and does not
intersection using formula (2): increase dramatically as the number of blocks increases
y=y1+(y2−y1)×((x−x1)/(x2−x1)) (2) because of the raster scanning method.

Find all the intersection and calculate all the 5.2 Experiment 2: Same number of blocks but
y-component values sorted in ascending; different fields
5) Make “intersection pairs” for the y-component In this experiment, the fields of different lengths
values as (y0, y1), (y2, y3), …, (yc−2, yc−1): (yi, i=0, 1, …, (56, 112, 168 bytes) were adopted and the number of
c−1) (c is number of intersections)); blocks model remained unchanged. We performed the
6) Set “internal” flag for the elements which center same comparative analysis as above. The results are
is located between one “intersection pairs”; shown in Table 2.
7) Execute (2) until all layers are processed. The results showed that the time-consuming of
As can be seen from the above, the algorithm does reading and writing increased with the increase of the
not have to calculate the intersection, just make a simple amount of data. The time-consuming of screening
judgment. As a result, the performance and robustness of remained about the same because of screening is only
the algorithm are improved greatly. related to number of block to be queried.
Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468 2467

Fig. 8 Mining and query region: (a) Structure model of ore; (b) Structure model of query region; (c) Block model of ore; (d) Block
model of query region

Table 1 Results of same fields but different numbers of blocks octants of an octree in increasing location code and the
Number of DB size/ Time consumed/s order of octants is exactly the same as that processed by
DB the preorder traversal of the octree, and the preorder
block MB Create Query Screen
traversal of the octree can track “z” pattern (z-order)
Etree 4.169 0.022 0.011
42517 0.637 [23,24], as shown in Fig. 9, and z curve has a good space
LMDB 4.968 0.033 0.005 accumulation [25].
Etree 32.689 0.549 0.184
339265 0.687
LMDB 39.136 0.316 0.061 Table 3 Result of different key-comparator functions
Etree 261.421 7.691 2.373 Comparison Time consumed/s
2715545 0.928 DB DB size/MB
LMDB 312.968 3.654 0.601 type Create Query
Note: N of block, the number of block; DB, the storage methods Etree 913.009 20.374 4.176
Memcmp
LMDB 942.449 26.750 0.840
Table 2 Results of same number of blocks but different fields
Etree 913.009 12.319 0.834
Field DB size/ Time consumed/s Integer-wise
DB LMDB 942.449 15.275 0.272
length/byte MB Create Query Screen
Etree 261.421 7.813 2.404
56 0.890
LMDB 312.968 3.772 0.602
Etree 484.305 12.338 3.327
112 0.887
LMDB 540.600 10.174 0.679
Etree 702.705 15.858 4.217
168 0.889
LMDB 774.476 13.213 0.749

5.3 Experiment 3: Different key-comparator


functions
In this experiment, both the length of the fields and
the number of blocks remain unchanged, but the
key-comparator function is different, one is integer-wise Fig. 9 Spatial aggregation of z-curve
comparison proposed previously, one is “memcmp”
which is a function comparing the first n characters of 6 Conclusions
two object. The results are shown in Table 3.
It is obvious that the key-comparator function 1) Owing to usage of memory-map and copy-on-
proposed in this work is better than memcmp function. write B+ tree, LMDB is superior to etree in storage of
The reason using this comparison function to sort the massive data of block model.
2468 Lin BI, et al/Trans. Nonferrous Met. Soc. China 26(2016) 2462−2468
2) Combining the advantages of linear octree and [11] TU T, O’HALLARON D R, LÓPEZ J C. Etree: A database-oriented
method for generating large octree meshes [J]. Engineering with
LMDB, the new method is a better way to realize the
Computers, 2004, 20(2): 117−28.
storage of massive data of geological attribute on 64 bit [12] CHU H. Mdb: A memory-mapped database and backend for
systems, and meets the characteristics of the database. openldap [J]. LDAP’11. 2011.
3) The performance and robustness of querying [13] KIM E, BIELAK J, GHATTAS O. Large-scale northridge earthquake
simulation using octree-based multiresolution mesh method
from block model are improved greatly by transforming [C]//Proceedings of the 16th ASCE Engineering Mechanics
from 3D to 2D and 2D grid scanning. Conference. Seattle: ASCE, 2003: 1−6.
[14] TU T. A scalable database approach to computing delaunay
triangulations [D]. Pittsburgh: Carnegie Mellon University, 2008.
References [15] GILL D, MAECHLING P J, JORDAN T H. UCVM: An open source
framework for 3D velocity model research [C]//AGU Fall Meeting
[1] HUA Wei-hua. Rapid modeling and quantitative analysis of the Abstracts, San Fracisco: AGU, 2013: 1612.
complex geological bodies with multi-constraint [D]. Wuhan: China [16] COMER D. Ubiquitous B-tree [J]. ACM Computing Surveys
University of Geosciences, 2010. (in Chinese) (CSUR), 1979, 11(2): 121−37.
[2] JIN B X, FANG Y M, SONG W W. 3D visualization model and key [17] CHU H. LIFE. After berkeley DB: OpenLDAP's memory-mapped
techniques for digital mine [J]. Transactions of Nonferrous Metals database [EB/OL]. [2012]. http://symas. com/mdb/20120829-
Society of China, 2011, 21(S3): s748−s752. LinuxCon-MDB-txt. pdf.
[3] WU Jiang-bin, ZHU He-hua. 3D ten model of strata and its [18] YADAVA H. The berkeley DB book [M]. New York: Apress, 2007.
realization based on delaunay triangulation [J]. Chinese Journal of [19] GU X W, WANG Q, SHU G E. Dynamic phase-mining optimization
Rock Mechanics and Engineering, 2005, 24(24): 4581−4587. in open-pit metal mines [J]. Transactions of Nonferrous Metals
[4] BI Lin, WANG Li-guan, CHEN Jian-hong. Study of octree-based Society of China, 2010, 20(10): 1974−1980.
block model of complex geological bodies [J]. China University of [20] JING Yong-bin, WANG Li-guan, BI Lin. Robust creation of block
Mining and Technology Journal, 2008, 37(4): 532−537. model from complex orebody model [J]. Journal of Huazhong
[5] CHERNYSHENKO A Y, OLSHANSKII M A. An adaptive octree University of Science and Technology (Natural Science Edition),
finite element method for PDEs posed on surfaces [J]. Computer 2010, 38(2): 97−100.
Methods in Applied Mechanics and Engineering, 2015, [21] LI Nan, WU Xin-cai, XIAO Ke-yan. A fast method for constructing
291:146−172. complex orebody block model of the algorithm [J]. Journal of
[6] HAN Guo-jian, GUO Da-zhi, JIN Xue-lin. Storage and indexing of Huazhong University of Science and Technology (Natural Science
orebody information using octrees [J]. Acta Geodaetica et Edition), 2013, 41(3): 34−37.
Cartographica Sinica, 1992, 21(1):13−17. [22] TAYLOR G. Point in polygon test [J]. Survey Review, 1994, 32(254):
[7] MORTON G M. A computer oriented geodetic data base and a new 479−84.
technique in file sequencing [M]. New York: International Business [23] ORENSTEIN J A, MERRETT T H. A class of data structures for
Machines Company, 1966. associative searching [C]//Proceedings of the 3rd ACM
[8] GARGANTINI I. An effective way to represent quadtrees [J]. SIGACT-SIGMOD symposium on Principles of Database Systems.
Communications of the ACM, 1982, 25(12): 905−910. Waterloo: ACM, 1984: 181−190.
[9] GARGANTINI I. Linear octrees for fast processing of three- [24] ORENSTEIN J A. Spatial query processing in an object-oriented
dimensional objects [J]. Computer Graphics and Image Processing. database system [J]. ACM Sigmod Record, 1986, 15(2): 326−336.
1982, 20(4): 365−374. [25] FALOUTSOS C, ROSEMAN S. Fractals for secondary key retrieval
[10] HAN J, HAIHONG E, LE G. Survey on NoSQL database [C]//Proceedings of the Eighth ACM SIGACT-SIGMOD-SIGART
[C]//Proceedings of 6th International Conference on Pervasive Symposium on Principles of Database Systems. ACM, 1989:
Computing and Applications (ICPCA), 2011. Albuquerque: IEEE, 247−252.
2011: 363−366.

面向数据库特征的基于 LMDB 与线性八叉树


海量块段模型存储技术
毕 林,赵 辉,贾明涛

中南大学 资源与安全工程学院,长沙 410083

摘 要:为满足数字矿山系统应用中对海量数据高效存取的技术要求,提出一种块段模型海量数据存储新方法。
该存储技术满足数据库的特点:ACID 兼容、并发支持、数据共享及高效访问;采用线性八叉树的方法组织块段
模型,并存将其储于 LMDB(快速内存映射数据库)中;通过定位码比较算法及从几何空间地址码到存储空间定位
码的转换算法,可高效地对三维空间任意点的地质属性进行查询;采用三维到二维的转换及内外点二维网格扫描
筛选法,使地质属性查询的三维问题转化为二维问题,其性能和鲁棒性得到显著提高。实验结果表明,这种方法
能够高效存取块段模型海量数据,并满足数据库的特点;相比于采用 etree 方法,采用 LMDB 方法至少快 3 倍,
特别是在读取数据时效率更高,且数据量越大,效果越明显。
关键词:块段模型;线性八叉树;快速内存映射数据库(LMDB);海量数据访问;数字矿山;etree
(Edited by Yun-bin HE)

You might also like