Column Oriented DB Systems
Content
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Part 1: Stavros Harizopoulos (HP Labs) Part 2: Daniel Abadi (Yale) Part 3: Peter Boncz (CWI)
VLDB 2009 Tutorial
VLDB 2009 Tutorial Column-Oriented Database Systems
1
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What is a column-store?
row-store
Date Store Product Customer Price
column-store
Date Store Product Customer Price
+ easy to add/modify a record - might read in unnecessary data
+ only need to read in relevant data - tuple writes require multiple accesses
=> suitable for read-mostly, read-intensive, large data repositories
VLDB 2009 Tutorial
Column-Oriented Database Systems
2
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Are these two fundamentally different?
l l
The only fundamental difference is the storage layout However: we need to look at the big picture
different storage layouts proposed row-stores row-stores++ ‘90s ‘00s row-stores++ today column-stores converge?
‘70s
‘80s
new applications new bottleneck in hardware
l l l
How did we get here, and where we are heading Part 1 What are the column-specific optimizations? Part 2 How do we improve CPU efficiency when operating on Cs
Part 3
VLDB 2009 Tutorial Column-Oriented Database Systems 3
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
4
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco Data Warehousing example
l
Typical DW installation Real-world example
dimension tables account or RAM usage toll source fact table
l
“One Size Fits All? - Part 2: Benchmarking Results” Stonebraker et al. CIDR 2007
QUERY 2 SELECT account.account_number, sum (usage.toll_airtime), sum (usage.toll_price) FROM usage, toll, source, account WHERE usage.toll_id = toll.toll_id AND usage.source_id = source.source_id AND usage.account_id = account.account_id AND toll.type_ind in (‘AE’. ‘AA’) AND usage.toll_price > 0 AND source.type != ‘CIBER’ AND toll.rating_method = ‘IS’ AND usage.invoice_date = 20051013 GROUP BY account.account_number
star schema
Query 1 Query 2 Query 3 Query 4 Query 5
Column-store 2.06 2.20 0.09 5.24 2.88
Row-store 300 300 300 300 300
Why? Three main factors (next slides)
Column-Oriented Database Systems 5
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (1/3): read efficiency
row store column store
read pages containing entire rows one row = 212 columns! is this typical? (it depends)
What about vertical partitioning? (it does not work with ad-hoc queries)
VLDB 2009 Tutorial
read only columns needed in this example: 7 columns caveats: • “select * ” not any faster • clever disk prefetching • clever tuple reconstruction
Column-Oriented Database Systems
6
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (2/3): compression efficiency
l
Columns compress better than rows
l l
Typical row-store compression ratio 1 : 3 Column-store 1 : 10
l
Why?
l
l l
Rows contain values from different domains => more entropy, difficult to dense-pack Columns exhibit significantly less entropy Male, Female, Female, Female, Male Examples:
1998, 1998, 1999, 1999, 1999, 2000
l
Caveat: CPU cost (use lightweight compression)
VLDB 2009 Tutorial
Column-Oriented Database Systems
7
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (3/3): sorting & indexing efficiency
l
Compression and dense-packing free up space
l l l l
Use multiple overlapping column collections Sorted columns compress better Range queries are faster Use sparse clustered indexes
What about heavily-indexed row-stores? (works well for single column access, cross-column joins become increasingly expensive)
VLDB 2009 Tutorial
Column-Oriented Database Systems
8
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Additional opportunities for column-stores
l
Block-tuple / vectorized processing
l
Easier to build block-tuple operators
l
Amortizes function-call cost, improves CPU cache performance Software-based: bitwise operations Hardware-based: SIMD Part 3
l
Easier to apply vectorized primitives
l l
l
Opportunities with compressed columns
l l
Avoid decompression: operate directly on compressed Delay decompression (and tuple reconstruction)
l
Also known as: late materialization
more in Part 2
l
Exploit columnar storage in other DBMS components
l
Physical design (both static and dynamic)
VLDB 2009 Tutorial Column-Oriented Database Systems
See: Database Cracking, from CWI
9
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Effect on C-Store performance
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Hachem, and Madden. SIGMOD 2008.
Average for SSBM queries on C-store
Time (sec)
original C-store
column-oriented join algorithm
enable compression & operate on compressed
enable late materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
10
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary of column-store key features
columnar storage header/ID elimination l Part 1 Part 3
Storage layout
compression
Part 2
multiple sort orders
column operators
Part 1
Part 2
l
Execution engine
avoid decompression late materialization Part 2 Part 3
vectorized operations l
Design tools, optimizer
VLDB 2009 Tutorial Column-Oriented Database Systems 11
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
12
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
From DSM to Column-stores
70s -1985:
TOD: Time Oriented Database – Wiederhold et al. "A Modular, Self-Describing Clinical Databank System," Computers and Biomedical Research, 1975 More 1970s: Transposed files, Lorie, Batory, Svensson. “An overview of cantor: a new system for data analysis” Karasalo, Svensson, SSDBM 1983 “A decomposition storage model” Copeland and Khoshafian. SIGMOD 1985.
1985: DSM paper
1990s: Commercialization through SybaseIQ Late 90s – 2000s: Focus on main-memory performance
l l
DSM “on steroids” [1997 – now] Hybrid DSM/NSM [2001 – 2004]
CWI: MonetDB Wisconsin: PAX, Fractured Mirrors CMU: Clotho
Michigan: Data Morphing MIT: C-Store
VLDB 2009 Tutorial
2005 – : Re-birth of read-optimized DSM as “column-store”
CWI: MonetDB/X100
Column-Oriented Database Systems
10+ startups
13
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
The original DSM paper
l l l l
“A decomposition storage model” Copeland and Khoshafian. SIGMOD 1985.
Proposed as an alternative to NSM 2 indexes: clustered on ID, non-clustered on value Speeds up queries projecting few columns Requires more storage value
ID
0100 0962 1000 .. 1 2 3 4 ..
VLDB 2009 Tutorial
Column-Oriented Database Systems
14
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory wall and PAX
l
90s: Cache-conscious research
“Cache Conscious Algorithms for from: Relational Query Processing.” Shatdal, Kant, Naughton. VLDB 1994. “Database Architecture Optimized for to: the New Bottleneck: Memory Access.” Boncz, Manegold, Kersten. VLDB 1999.
“DBMSs on a modern processor: and: Where does time go?” Ailamaki, DeWitt, Hill, Wood. VLDB 1999.
l
PAX: Partition Attributes Across
l l
Retains NSM I/O pattern Optimizes cache-to-RAM communication
“Weaving Relations for Cache Performance.” Ailamaki, DeWitt, Hill, Skounakis, VLDB 2001.
VLDB 2009 Tutorial
Column-Oriented Database Systems
15
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
More hybrid NSM/DSM schemes
l
Dynamic PAX: Data Morphing
“Data morphing: an adaptive, cache-conscious storage technique.” Hankins, Patel, VLDB 2003.
l
Clotho: custom layout using scatter-gather I/O
“Clotho: Decoupling Memory Page Layout from Storage Organization.” Shao, Schindler, Schlosser, Ailamaki, and Ganger. VLDB 2004.
l
Fractured mirrors
l
Smart mirroring with both NSM/DSM copies
“A Case For Fractured Mirrors.” Ramamurthy, DeWitt, Su, VLDB 2002.
VLDB 2009 Tutorial
Column-Oriented Database Systems
16
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB (more in Part 3)
l l
Late 1990s, CWI: Boncz, Manegold, and Kersten Motivation:
l l
l l
Main-memory Improve computational efficiency by avoiding expression interpreter DSM with virtual IDs natural choice Developed new query execution algebra Pointed out memory-wall in DBMSs Cache-conscious projections and joins …
VLDB 2009 Tutorial Column-Oriented Database Systems 17
l
Initial contributions:
l l l
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
2005: the (re)birth of column-stores
l
New hardware and application realities
l l
Faster CPUs, larger memories, disk bandwidth limit Multi-terabyte Data Warehouses Read-optimized, fast multi-column access, disk/CPU efficiency, light-weight compression
l
New approach: combine several techniques
l
l
C-store paper:
l
First comprehensive design description of a column-store “proper” disk-based column store
l
MonetDB/X100
l
l
Explosion of new products
VLDB 2009 Tutorial Column-Oriented Database Systems 18
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Performance tradeoffs: columns vs. rows
DSM traditionally was not favored by technology trends How has this changed?
l l
Optimized DSM in “Fractured Mirrors,” 2002 “Apples-to-apples” comparison “Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
l l
Follow-up study
“Read-Optimized Databases, InDepth” Holloway, DeWitt, VLDB’08
Main-memory DSM vs. NSM
“DSM vs. NSM: CPU performance tradeoffs in block-oriented query processing” Boncz, Zukowski, Nes, DaMoN’08
l
Flash-disks: a come-back for PAX?
“Query Processing Techniques “Fast Scans and Joins Using Flash for Solid State Drives” Drives” Shah, Harizopoulos, Tsirogiannis, Harizopoulos, Wiener, Graefe. DaMoN’08 Shah, Wiener, Graefe, VLDB 2009 Tutorial Column-Oriented Database Systems 19 SIGMOD’09
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Fractured mirrors: a closer look
l
Store DSM relations inside a B-tree
l l l
Tuple Header
Leaf nodes contain values Eliminate IDs, amortize header overhead Custom implementation on Shore
TID Column Data
“A Case For Fractured Mirrors” Ramamurthy, DeWitt, Su, VLDB 2002.
sparse B-tree on ID 3
1 2 3 4 5
a1 a2 a3 a4 a5
1
a1 1
2 a1
a2
3
a3
4 4
a4
5
a5
a2 a3
a4 a5
Similar: storage density “Efficient columnar comparable storage in B-trees” Graefe. to column stores Sigmod Record 03/2007.
VLDB 2009 Tutorial Column-Oriented Database Systems 20
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Fractured mirrors: performance
From PAX paper: time regular DSM column? column? row
columns projected: 1 2 3 4 5
l
Chunk-based tuple merging
l l l
optimized DSM
Read in segments of M pages Merge segments in memory Becomes CPU-bound after 5 pages
VLDB 2009 Tutorial Column-Oriented Database Systems 21
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-scanner implementation
row scanner
“Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
column scanner
Joe 45 … …
Joe 45 … …
SELECT name, age WHERE age > 40
apply predicate(s)
S
Direct I/O Joe Sue … prefetch ~100ms worth of data
S
#POS 45 #POS …
1 Joe 45 2 Sue 37 …… …
apply predicate #1 45 37 …
S
VLDB 2009 Tutorial
Column-Oriented Database Systems
22
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Scan performance
l l l l l
Large prefetch hides disk seeks in columns Column-CPU efficiency with lower selectivity Row-CPU suffers from memory stalls Memory stalls disappear in narrow tuples Compression: similar to narrow
not shown, details in the paper
VLDB 2009 Tutorial
Column-Oriented Database Systems
23
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Even more results
• •
“Read-Optimized Databases, InDepth” Holloway, DeWitt, VLDB’08
35
Same engine as before Additional findings
narrow & compressed tuple: 30 CPU-bound!
25
C-25% C-10% R-50%
Time (s)
20
15
10
wide attributes: same as before
5
0 1 2 3 4 5 6 7 Columns Returned 8 9 10
Non-selective queries, narrow tuples, favor well-compressed rows Materialized views are a win Column-joins are Scan times determine early materialized joins covered in part 2!
VLDB 2009 Tutorial Column-Oriented Database Systems 24
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Speedup of columns over rows
cycles per disk byte
144 72 36 18 9 “Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
(cpdb)
+++
_ = + ++
8 12 16 20 24 28 32 36
tuple width
l
Rows favored by narrow tuples and low cpdb
l
Disk-bound workloads have higher cpdb
VLDB 2009 Tutorial Column-Oriented Database Systems 25
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Varying prefetch size
no competing disk traffic
40
time (sec)
Column 2
30 20 10 0 4 8 12 16 20 24 28 32
Column 8 Column 16 Column 48 (x 128KB) Row (any prefetch size)
selected bytes per tuple
l
No prefetching hurts columns in single scans
VLDB 2009 Tutorial Column-Oriented Database Systems 26
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Varying prefetch size
with competing disk traffic
40
time (sec)
30 20 10 0 4
Column, 48 Row, 48
40 30 20 10 0 Column, 8 Row, 8 4 12 20 28
12
20
28
selected bytes per tuple
l l
No prefetching hurts columns in single scans Under competing traffic, columns outperform rows for any prefetch size
VLDB 2009 Tutorial Column-Oriented Database Systems 27
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Performance
l l l
“DSM vs. NSM: CPU performance trade offs in block-oriented query processing” Boncz, Zukowski, Nes, DaMoN’08
Benefit in on-the-fly conversion between NSM and DSM DSM: sequential access (block fits in L2), random in L1 NSM: random access, SIMD for grouped Aggregation
VLDB 2009 Tutorial
Column-Oriented Database Systems
28
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
New storage technology: Flash SSDs
l
Performance characteristics
l l
very fast random reads, slow random writes fast sequential reads and writes
l
Price per bit (capacity follows)
l
cheaper than RAM, order of magnitude more expensive than Disk
avoid random writes! SSD (Ł small reads still suffer from SATA overhead/OS limitations) PCI card (Ł high price, limited expandability)
l
Flash Translation Layer introduces unpredictability
l
l
Form factors not ideal yet
l l
l
Boost Sequential I/O in a simple package
l
Flash RAID: very tight bandwidth/cm3 packing (4GB/sec inside the box) useful for delta structures and logs still suboptimal if I/O block size > record size therefore column stores profit mush less than horizontal stores the larger the data, the deeper clustering one can exploit
Column-Oriented Database Systems 29
l
Column Store Updates
l
l
Random I/O on flash fixes unclustered index access
l l
l
Random I/O useful to exploit secondary, tertiary table orderings
l VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Even faster column scans on flash SSDs
l
New-generation SSDs
l l
30K Read IOps, 3K Write Iops 250MB/s Read BW, 200MB/s Write
Very fast random reads, slower random writes Fast sequential RW, comparable to HDD arrays
l l
No expensive seeks across columns FlashScan and Flashjoin: PAX on SSDs, inside Postgres
“Query Processing Techniques for Solid State Drives” Tsirogiannis, Harizopoulos, Shah, Wiener, Graefe, SIGMOD’09 mini-pages with no qualified attributes are not accessed
VLDB 2009 Tutorial
Column-Oriented Database Systems
30
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-scan performance over time
regular DSM (2001) from 7x slower column-store (2006) ..to 1.2x slower
..to 2x slower
..to same
and 3x faster! optimized DSM (2002)
VLDB 2009 Tutorial
SSD Postgres/PAX (2009)
Column-Oriented Database Systems 31
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
32
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Architecture of a column-store
storage layout
l l l l
read-optimized: dense-packed, compressed organize in extends, batch updates multiple sort orders engine sparse indexes l block-tuple operators l new access methods l optimized relational operators system-level system-wide column support loading / updates scaling through multiple nodes transactions / redundancy
VLDB 2009 Tutorial Column-Oriented Database Systems 33
l l l l
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
C-Store
l l l l l l l l l l l
“C-Store: A Column-Oriented DBMS.” Stonebraker et al. VLDB 2005.
Compress columns No alignment Big disk blocks Only materialized views (perhaps many) Focus on Sorting not indexing Data ordered on anything, not just time Automatic physical DBMS design Optimize for grid computing Innovative redundancy Xacts – but no need for Mohan Column optimizer and executor
VLDB 2009 Tutorial Column-Oriented Database Systems 34
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
C-Store: only materialized views (MVs)
l l
l l l l l
Projection (MV) is some number of columns from a fact table Plus columns in a dimension table – with a 1-n join between Fact and Dimension table Stored in order of a storage key(s) Several may be stored! With a permutation, if necessary, to map between them Table (as the user specified it and sees it) is not stored! No secondary indexes (they are a one column sorted MV plus a permutation, if you really want one)
User view:
EMP (name, age, salary, dept) Dept (dname, floor)
Possible set of MVs:
MV-1 (name, dept, floor) in floor order MV-2 (salary, age) in age order MV-3 (dname, salary, name) in salary order
Column-Oriented Database Systems 35
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Continuous Load and Query (Vertica)
Hybrid Storage Architecture
> Write Optimized Store (WOS)
> Read Optimized Store (ROS)
TUPLE MOVER • On disk • Sorted / Compressed • Segmented • Large data loaded direct
Trickle Load
A B C
Asynchronous Data Transfer
§Memory based §Unsorted / Uncompressed §Segmented §Low latency / Small quick inserts
VLDB 2009 Tutorial Column-Oriented Database Systems
A
B
C
(A B C | A)
36
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Loading Data (Vertica)
> INSERT, UPDATE, DELETE > Bulk and Trickle Loads
Write-Optimized Store (WOS) In-memory Automatic Tuple Mover
§COPY §COPY DIRECT
> User loads data into logical Tables > Vertica loads atomically into storage
Read-Optimized Store (ROS) On-disk
VLDB 2009 Tutorial
Column-Oriented Database Systems
37
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Applications for column-stores
l
Data Warehousing
l l l
High end (clustering) Mid end/Mass Market Personal Analytics E.g. Proximity
l
Data Mining
l
l l
Google BigTable RDF
l
Semantic web data management Terabyte TREC SciDB initiative SLOAN Digital Sky Survey on MonetDB
l
Information retrieval
l
l
Scientific datasets
l l
VLDB 2009 Tutorial
Column-Oriented Database Systems
38
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
List of column-store systems
l l l l l l l l l l l l l
Cantor (history) Sybase IQ SenSage (former Addamark Technologies) Kdb 1010data MonetDB C-Store/Vertica X100/VectorWise KickFire SAP Business Accelerator Infobright ParAccel Exasol
VLDB 2009 Tutorial Column-Oriented Database Systems 39
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
40
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Simulate a Column-Store inside a Row-Store
Date Store Product Customer Price
01/01 BOS Table 01/01 NYC Chair 01/01 BOS Bed
Mesa Lutz Mudd
$20 $13 $79
Option B: Index Every Column
Date Index
Option A: Vertical Partitioning
Date
TID Value
Store
TID Value
Product
TID Value
Customer
TID Value TID
Price
Value
1 01/01 2 01/01 3 01/01
1 2 3
BOS NYC BOS
1 Table 2 Chair 3 Bed
1 2 3
Mesa Lutz Mudd
1 2 3
$20 $13 $79
Store Index
…
VLDB 2009 Tutorial Column-Oriented Database Systems 41
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Simulate a Column-Store inside a Row-Store
Date Store Product Customer Price
01/01 BOS Table 01/01 NYC Chair 01/01 BOS Bed
Mesa Lutz Mudd
$20 $13 $79
Option B: Index Every Column
Date Index
Option A: Vertical Partitioning
Date
Value StartPos Length
Store
TID Value
Product
TID Value
Customer
TID Value TID
Price
Value
01/01
1
3
Can explicitly runlength encode date
1 2 3
BOS NYC BOS
1 Table 1 2 Chair 2 3 Bed 3
Mesa Lutz Mudd
1 2 3
$20 $13 $79
Store Index
“Teaching an Old Elephant New Tricks.” Bruno, CIDR 2009.
VLDB 2009 Tutorial Column-Oriented Database Systems
…
42
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Experiments
l
Star Schema Benchmark (SSBM)
Adjoined Dimension Column Index (ADC Index) to Improve Star Schema Query Performance”. O’Neil et. al. ICDE 2008.
l l
Implemented by professional DBA Original row-store plus 2 column-store simulations on same row-store product
250.0 200.0
Time (seconds)
150.0 100.0 50.0 0.0
Normal Row-Store Average 25.7
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Hachem, and Madden. SIGMOD 2008.
Vertically Partitioned Row-Store 79.9
Row-Store With All Indexes 221.2
VLDB 2009 Tutorial
Column-Oriented Database Systems
43
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What’s Going On? Vertical Partitions
l
Vertical partitions in row-stores:
l l l
Work well when workload is known ..and queries access disjoint sets of columns See automated physical design
Tuple Header TID Column Data
l
Do not work well as full-columns
l l
TupleID overhead significant Excessive joins
1 2 3
Queries touch 3-4 foreign keys in fact table, 1-2 numeric columns
“Column-Stores vs. Row-Stores: How Different Are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
VLDB 2009 Tutorial
Complete fact table takes up ~4 GB (compressed) Vertically partitioned tables take up 0.7-1.1 GB (compressed)
Column-Oriented Database Systems 44
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What’s Going On? All Indexes Case
l
Tuple construction
l
Common type of query:
SELECT store_name, SUM(revenue) FROM Facts, Stores WHERE fact.store_id = stores.store_id AND stores.country = “Canada” GROUP BY store_name
l
l
Result of lower part of query plan is a set of TIDs that passed all predicates Need to extract SELECT attributes at these TIDs
l l
BUT: index maps value to TID You really want to map TID to value (i.e., a vertical partition)
Tuple construction is SLOW
VLDB 2009 Tutorial Column-Oriented Database Systems 45
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
So….
l l
All indexes approach is a poor way to simulate a column-store Problems with vertical partitioning are NOT fundamental
l l l
Store tuple header in a separate partition Allow virtual TIDs Combine clustered indexes, vertical partitioning Might be possible, BUT: l Need better support for vertical partitioning at the storage layer l Need support for column-specific optimizations at the executer level l Full integration: buffer pool, transaction manager, ..
l
So can row-stores simulate column-stores?
l
l
When will this happen?
l
Most promising features = soon
See Part 2, Part 3 for most promising features
l
..unless new technology / new objectives change the game (SSDs, Massively Parallel Platforms, Energy-efficiency)
Column-Oriented Database Systems 46
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
End of Part 1
l
Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
47
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Part 2 Outline
l
Compression Tuple Materialization Joins
l
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
48
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Compression
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06 “Integrating Compression and Execution in ColumnOriented Database Systems” Abadi, Madden, and Ferreira, SIGMOD ’06 •Query optimization in compressed database systems” Chen, Gehrke, Korn, SIGMOD’01
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Compression
l l
Trades I/O for CPU Increased column-store opportunities:
l l
l
Higher data value locality in column stores Techniques such as run length encoding far more useful Can use extra space to store multiple copies of data in different sort orders
VLDB 2009 Tutorial
Column-Oriented Database Systems
50
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Run-length Encoding
Quarter Product ID Price Q1 Q1 Q1 Q1 Q1 Q1 Q1 … Q2 Q2 Q2 Q2 … 1 1 1 1 1 2 2 … 1 1 1 2 … 5 7 2 9 6 8 5 … 3 8 1 4 …
Column-Oriented Database Systems
Quarter
(value, start_pos, run_length)
Product ID (1, 1, 5) (2, 6, 2) …
Price 5 7 2 9 6 8 5 … 3 8 1 4 …
51
(value, start_pos, run_length)
(Q1, 1, 300) (Q2, 301, 350) (Q3, 651, 500) (Q4, 1151, 600)
(1, 301, 3) (2, 304, 1) …
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Bit-vector Encoding
l
For each unique Product ID value, v, in column 1 c, create bit-vector 1 b
l
ID: 1 1 1 1 1 1 0 0 … 1 1 0 0 …
ID: 2 0 0 0 0 0 1 1 … 0 0 1 0 …
ID: 3 0 0 0 0 0 0 0 … 0 0 0 1 …
… 0 0 0 0 0 0 0 … 0 0 0 0 …
52
b[i] = 1 if c[i] = v
l
l
Good for columns with few unique values Each bit-vector can be further compressed if sparse
1 1 1 2 2 … 1 1 2 3 …
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Dictionary Encoding
l
l
l
For each unique value create dictionary entry Dictionary can be per-block or per-column Column-stores have the advantage that dictionary entries may encode multiple values at once
Quarter Quarter Q1 Q2 Q4 Q1 Q3 Q1 Q1 Q1 Q2 Q4 Q3 Q3 …
0 1 3 0 2 0 0 0 1 3 2 2
Quarter 24 128 122
+
Dictionary 0: Q1 1: Q2 2: Q3 3: Q4
OR
+
Dictionary 24: Q1, Q2, Q4, Q1 … 122: Q2, Q4, Q3, Q3 … 128: Q3, Q1, Q1, Q1
53
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Frame Of Reference Encoding
l
l
Encodes values as b bit offset from chosen frame of reference Special escape code (e.g. all bits set to 1) indicates a difference larger than can be stored in b bits
l
Price 45 54 48 55 51 53 40 50 49 62 52 50 …
Price
Frame: 50
After escape code, original (uncompressed) value is written
“Compressing Relations and Indexes ” Goldstein, Ramakrishnan, Shaft, ICDE’98
VLDB 2009 Tutorial
-5 4 -2 5 1 3 ∞ 40 0 -1 ∞ 62 2 0 …
4 bits per value
Exceptions (see part 3 for a better way to deal with exceptions)
Column-Oriented Database Systems
54
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Differential Encoding
l l
Encodes values as b bit offset from previous value Special escape code (just like frame of reference encoding) indicates a difference larger than can be stored in b bits
l
Time 5:00 5:02 5:03 5:03 5:04 5:06 5:07 5:08 5:10 5:15 5:16 5:16 …
Time 5:00 2 1 0 1 2 1 1 2 ∞ 5:15 1 0
After escape code, original (uncompressed) value is written
2 bits per value
l
Performs well on columns containing increasing/decreasing sequences
l l l l
inverted lists timestamps object IDs sorted / clustered columns
Exception (see part 3 for a better way to deal with exceptions)
“Improved Word-Aligned Binary Compression for Text Indexing” Ahn, Moffat, TKDE’06
VLDB 2009 Tutorial
Column-Oriented Database Systems
55
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What Compression Scheme To Use?
Does column appear in the sort key? yes Is the average run-length > 2 yes RLE no Differential Encoding yes no Does this column appear frequently in selection predicates? no Are number of unique values < ~50000
Is the data numerical and exhibit good locality? yes Frame of Reference Encoding no Leave Data Uncompressed
yes
no
Bit-vector Compression
Dictionary Compression
OR
Heavyweight Compression
VLDB 2009 Tutorial
Column-Oriented Database Systems
56
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Heavy-Weight Compression Schemes
l l
Ł Ł
Modern disk arrays can achieve > 1GB/s 1/3 CPU for decompression Ł 3GB/s needed
Lightweight compression schemes are better Even better: operate directly on compressed data
VLDB 2009 Tutorial Column-Oriented Database Systems 57
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
l l l
I/O - CPU tradeoff is no longer a tradeoff Reduces memory–CPU bandwidth requirements Opens up possibility of operating on multiple records at once
VLDB 2009 Tutorial
Column-Oriented Database Systems
58
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
Quarter Product ID 1 … (Q1, 1, 300) (Q2, 301, 6) (Q3, 307, 500) (Q4, 807, 600)
301-306
2 … 0 0 1 0 0 0 1 0 0 1 0 …
3 … 0 1 0 0 0 1 0 0 1 0 1 …
ProductID, COUNT(*))
Index Lookup + Offset jump
1 0 0 1 1 0 0 1 0 0 0 …
(1, 3) (2, 1) (3, 2) SELECT ProductID, Count(*) FROM table WHERE (Quarter = Q2) GROUP BY ProductID
VLDB 2009 Tutorial
Column-Oriented Database Systems
59
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
Block API Data Aggregation Operator isOneValue() isValueSorted() isPosContiguous() isSparse() getNext() decompressIntoArray() getValueAtPosition(pos) getMin() getMax() getSize() (Q1, 1, 300) (Q2, 301, 6) (Q3, 307, 500) (Q4, 807, 600)
SELECT ProductID, Count(*) FROM table WHERE (Quarter = Q2) GROUP BY ProductID
Selection Operator
CompressionAware Scan Operator
VLDB 2009 Tutorial
Column-Oriented Database Systems
60
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Tuple Materialization and Column-Oriented Join Algorithms
“Materialization Strategies in a ColumnOriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007. “Self-organizing tuple reconstruction in column-stores“, Idreos, Manegold, Kersten, SIGMOD’09 “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008. “Query Processing Techniques for Solid State Drives” Tsirogiannis, Harizopoulos Shah, Wiener, and Graefe. SIGMOD 2009. “Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
When should columns be projected?
l
Where should column projection operators be placed in a query plan?
l
Row-store:
l l
Column projection involves removing unneeded columns from tuples Generally done as early as possible Operation is almost completely opposite from a row-store Column projection involves reading needed columns from storage and extracting values for a listed set of tuples
§
l
Column-store:
l l
This process is called “materialization”
Straightforward since there is a one-to-one mapping across columns More complicated since selection and join operators on one column obfuscates mapping to other columns from same table Many database interfaces expect output in regular tuples (rows) Rest of discussion will focus on this case
l l
Early materialization: project columns at beginning of query plan
§
Late materialization: wait as long as possible for projecting columns
§
l
Most column-stores construct tuples and column projection time
§ §
VLDB 2009 Tutorial
Column-Oriented Database Systems
62
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
When should tuples be constructed?
Select + Aggregate 4 4 4 4 2 1 3 1 2 7 3 13 3 42 3 80
l
QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
Construct (4,1,4) 2 1 3 1
storeID
Solution 1: Create rows first (EM). But:
l l l
2 3 3 3
7 13 42 80
Need to construct ALL tuples Need to decompress data Poor memory bandwidth utilization
prodID
custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
63
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
1 1 1 1
Data Source price Data Source Data Source
AGG
Data Source custID
0 1 0 1
QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND 4 4 4 4
prodID
Data Source prodID
Data Source storeID
2 1 3 1
storeID
4 4 4 4
2 1 3 1
2 3 3 3
7 13 42 80
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
64
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG
Data Source custID Data Source price
0 1 0 1
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND AND 1 1 1 1 0 1 0 1 4 4 4 4 2 1 3 1 2 3 3 3 7 13 42 80
Data Source prodID
Data Source storeID
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
65
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG
Data Source custID Data Source price
3 3 2 0 3 1 3 0 3 1
custID
13 1 80 1 7 0 13 1 42 0 80 1
price
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
Data Source
Data Source
AND
Data Source prodID
Data Source storeID
0 1 0 1
4 4 4 4
2 1 3 1
2 3 3 3
7 13 42 80
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
66
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG 3 1 1 93
Data Source custID Data Source price
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND
AGG 4 4 4 4 2 1 3 1 2 3 3 3 7 13 42 80
Data Source prodID
Data Source storeID
3 1 3 1
13 1 80 1
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
67
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Materialization Strategies in a Column-Oriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007.
For plans without joins, late materialization is a win
10 9 8 7 6 5 4 3 2 1 0 Low selectivity Medium selectivity High selectivity
QUERY: SELECT C1, SUM(C2) FROM table WHERE (C1 < CONST) AND (C2 < CONST) GROUP BY C1
l
Time (seconds)
Ran on 2 compressed columns from TPC-H scale 10 data
Early Materialization
Late Materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
68
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Materialization Strategies in a Column-Oriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007.
Even on uncompressed data, late materialization is still a win
14 12
QUERY: SELECT C1, SUM(C2) FROM table WHERE (C1 < CONST) AND (C2 < CONST) GROUP BY C1
Time (seconds)
10 8 6 4 2 0 Low selectivity Medium selectivity High selectivity
l
Materializing late still works best
Early Materialization
Late Materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
69
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What about for plans with joins?
Select R1.B, R1.C, R2.E, R2.H, R3.F From R1, R2, R3 Where R1.A = R2.D AND R2.G = R3.K B, C, E, H, F B, C
Join 2 G=K
B, C, E, H, F
Project Join G=K
F K
Scan
G F, K
Project Join A=D
B, C, E, G, H
Join 1 A=D
R3 (F, K, L) Scan R3 (F, K, L)
G D
Scan
E, H
A, B, C
Scan
D, E, G, H
Scan
A
Scan
Column-Oriented Database Systems
R1 (A, B, C) R2 (D, E, G, H)
VLDB 2009 Tutorial
R1 (A, B, C) R2 (D, E, G, H)
70
70
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What about for plans with joins?
Select R1.B, R1.C, R2.E, R2.H, R3.F From R1, R2, R3 Where R1.A = R2.D AND R2.G = R3.K B, C, E, H, F B, C
Join 2 G=K
B, C, E, H, F
Project Join G=K
F K
Scan
G F, K
B, C, E, G, H Early
Late Project
Join A=D
materialization Join 1
A=D Scan R3 (F, K, L)
materialization (F, K, L) R3
G D
Scan
71
E, H
A, B, C
Scan
D, E, G, H
Scan
A
Scan
Column-Oriented Database Systems
R1 (A, B, C) R2 (D, E, G, H)
VLDB 2009 Tutorial
R1 (A, B, C) R2 (D, E, G, H)
71
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Early Materialization Example
2 7 3 13 3 42 3 80 Construct (4,1,4) 12 1 11 1 6 1 2 1 2 3 3 3 7 13 42 80 1 Green 2 White 3 Brown Construct
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
1 2 3
custID
Green White Brown
lastName
prodID storeID quantity custID price
Facts
VLDB 2009 Tutorial
Customers
Column-Oriented Database Systems 72
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Early Materialization Example
7 White
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
13 Brown 42 Brown 80 Brown Join 2 7 3 13 3 42 3 80 1 Green 2 White 3 Brown
VLDB 2009 Tutorial
Column-Oriented Database Systems
73
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Late Materialization Example
1 2 3 4 Join 2 3 1 3
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
(4,1,4)
12 1 11 1
6 1 2 1
2 3 1 3
7 13 42 80
1 2 3
custID
Green White Brown
lastName
Late materialized join causes out of order probing of projected columns from the inner relation
prodID
storeID quantity custID price
Facts
VLDB 2009 Tutorial
Customers
Column-Oriented Database Systems 74
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Late Materialized Join Performance
l
Naïve LM join about 2X slower than EM join on typical queries (due to random I/O)
l
This number is very dependent on
l l l
Amount of memory available Number of projected attributes Join cardinality
l
But we can do better
l l l
Invisible Join Jive/Flash Join Radix cluster/decluster join
VLDB 2009 Tutorial
Column-Oriented Database Systems
75
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
l
Designed for typical joins when data is modeled using a star schema
l
One (“fact”) table is joined with multiple dimension tables select c_nation, s_nation, d_year, sum(lo_revenue) as revenue from customer, lineorder, supplier, date where lo_custkey = c_custkey and lo_suppkey = s_suppkey and lo_orderdate = d_datekey and c_region = 'ASIA‘ and s_region = 'ASIA‘ and d_year >= 1992 and d_year <= 1997 group by c_nation, s_nation, d_year order by d_year asc, revenue desc;
l
Typical query:
VLDB 2009 Tutorial
Column-Oriented Database Systems
76
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
custkey region nation 1 ASIA CHINA 2 ASIA INDIA 3 ASIA INDIA 4 EUROPE FRANCE
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
Apply “region = ‘Asia’” On Customer Table
… … … … …
Hash Table (or bit-map) Containing Keys 1, 2 and 3
Apply “region = ‘Asia’” On Supplier Table
suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 ASIA JAPAN … … … …
Hash Table (or bit-map) Containing Keys 1, 3
Apply “year in [1992,1997]” On Date Table
dateid 01011997 01021997 01031997 year 1997 1997 1997 … … … …
Hash Table Containing Keys 01011997, 01021997, and 01031997
Column-Oriented Database Systems 77
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Original Fact Table
orderkey custkey suppkey orderdate revenue 1 3 1 01011997 43256 2 3 2 01011997 33333 3 4 3 01021997 12121 4 1 1 01021997 23233 5 4 2 01021997 45456 6 1 2 01031997 43251 7 3 2 01031997 34235
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi et. al. SIGMOD 2008. 1 1 1 1 0 1 0 1 1 1 1 1 0 0 1 1 0 1 1 0 1
& & = +
1 0 0 1 0 0 0
Hash Table Containing Keys 1, 2 and 3
custkey 3 3 4 1 4 1 3
VLDB 2009 Tutorial
+
Hash Table Containing Keys 1 and 3
1 1 0 1 0 1 1
suppkey 1 2 3 1 2 2 2
+
Hash Table Containing Keys 01011997, 01021997, and 01031997
1 0 1 1 0 0 0
orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
1 1 1 1 1 1 1
78
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
custkey 3 3 4 1 4 1 3 suppkey 1 2 3 1 2 2 2 orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
+ + +
1 0 0 1 0 0 0
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
1997 1997 1997
INDIA CHINA
1 0 0 1 0 0 0
1 1
RUSSIA SPAIN JAPAN
RUSSIA RUSSIA
1 0 0 1 0 0 0
01011997 01021997
JOIN
01011997 01021997 01031997
=
1997 1997
VLDB 2009 Tutorial
Column-Oriented Database Systems
79
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
custkey region nation 1 ASIA CHINA 2 ASIA INDIA 3 ASIA INDIA 4 EUROPE FRANCE
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
Apply “region = ‘Asia’” On Customer Table
… … … … …
Hash Table (or bit-map) Containing Keys 1, 2 and 3
Range [1-3]
(between-predicate rewriting)
Apply “region = ‘Asia’” On Supplier Table
suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 ASIA JAPAN … … … …
Hash Table (or bit-map) Containing Keys 1, 3
Apply “year in [1992,1997]” On Date Table
dateid 01011997 01021997 01031997 year 1997 1997 1997 … … … …
Hash Table Containing Keys 01011997, 01021997, and 01031997
Column-Oriented Database Systems 80
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
l
Bottom Line
l
l
l
l
l
l
Many data warehouses model data using star/snowflake schemes Joins of one (fact) table with many dimension tables is common Invisible join takes advantage of this by making sure that the table that can be accessed in position order is the fact table for each join Position lists from the fact table are then intersected (in position order) This reduces the amount of data that must be accessed out of order from the dimension tables “Between-predicate rewriting” trick not relevant for this discussion
Column-Oriented Database Systems 81
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
custkey 3 3 4 1 4 1 3 suppkey 1 2 3 1 2 2 2 orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
+ + +
1 0 0 1 0 0 0
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
1997 1997 1997
INDIA CHINA
Still accessing table out of order
1 0 0 1 0 0 0 1 1
RUSSIA SPAIN JAPAN
RUSSIA RUSSIA
1 0 0 1 0 0 0
01011997 01021997
JOIN
01011997 01021997 01031997
=
1997 1997
VLDB 2009 Tutorial
Column-Oriented Database Systems
82
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
3 1
+
CHINA INDIA INDIA FRANCE
=
INDIA CHINA
“Fast Joins using Join Indices”. Li and Ross, VLDBJ 8:1-24, 1999.
Still accessing table out of order
“Query Processing Techniques for Solid State Drives”. Tsirogiannis, Harizopoulos et. al. SIGMOD 2009.
VLDB 2009 Tutorial
Column-Oriented Database Systems
83
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
1.
2.
3.
4.
Add column with dense ascending integers from 1 Sort new position list by second column Probe projected column in order using new sorted position list, keeping first column from position list around Sort new result by first column
1 2
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
2 1
INDIA CHINA
2 1
1 3
CHINA INDIA INDIA FRANCE
CHINA INDIA
1 2
INDIA CHINA
VLDB 2009 Tutorial
Column-Oriented Database Systems
84
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
l
Bottom Line
l
Instead of probing projected columns from inner table out of order:
l l l
Sort join index Probe projected columns in order Sort result using an added column LM has the extra sorts (EM accesses all columns in order) LM only has to fit join columns into memory (EM needs join columns and all projected columns)
§
l
LM vs EM tradeoffs:
l l
Results in big memory and CPU savings (see part 3 for why there is CPU savings)
l l
LM only has to materialize relevant columns In many cases LM advantages outweigh disadvantages
l
LM would be a clear winner if not for those pesky sorts … can we do better?
VLDB 2009 Tutorial
Column-Oriented Database Systems
85
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Radix Cluster/Decluster
l
The full sort from the Jive join is actually overkill
l
l l
We just want to access the storage blocks in order (we don’t mind random access within a block) So do a radix sort and stop early By stopping early, data within each block is accessed out of order, but in the order specified in the original join index
l
Use this pseudo-order to accelerate the post-probe sort as well
•“Database Architecture Optimized for the New Bottleneck: Memory Access” VLDB’99 •“Generic Database Cost Models for Hierarchical Memory Systems”, VLDB’02 (all Manegold, Boncz, Kersten)
“Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
VLDB 2009 Tutorial
Column-Oriented Database Systems
86
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Radix Cluster/Decluster
l
Bottom line
l
l
Both sorts from the Jive join can be significantly reduced in overhead Only been tested when there is sufficient memory for the entire join index to be stored three times
l
Technique is likely applicable to larger join indexes, but utility will go down a little Don’t want to use radix cluster/decluster if you have variablewidth column values or compression schemes that can only be decompressed starting from the beginning of the block
l
Only works if random access within a storage block
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
87
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
LM vs EM joins
l
l
Invisible, Jive, Flash, Cluster, Decluster techniques contain a bag of tricks to improve LM joins Research papers show that LM joins become 2X faster than EM joins (instead of 2X slower) for a wide array of query types
VLDB 2009 Tutorial
Column-Oriented Database Systems
88
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Tuple Construction Heuristics
l
l
For queries with selective predicates, aggregations, or compressed data, use late materialization For joins:
l
Research papers:
l
Always use late materialization Inner table to a join often materialized before join (reduces system complexity): Some systems will use LM only if columns from inner table can fit entirely in memory
l
Commercial systems:
l
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
89
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
l l
Computational Efficiency of DB on modern hardware
how column-stores can help here Keynote revisited: MonetDB & VectorWise in more depth
l
l
CPU efficient column compression
vectorized decompression
l
l
Conclusions
future work
VLDB 2009 Tutorial
Column-Oriented Database Systems
90
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
40 years of hardware evolution vs. DBMS computational efficiency
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Architecture
Elements: l Storage
l
CPU caches L1/L2/L3
l l
Registers Execution Unit(s)
l l
Pipelined SIMD
VLDB 2009 Tutorial
Column-Oriented Database Systems
92
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
93
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
94
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DRAM Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
95
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Super-Scalar Execution (pipelining)
VLDB 2009 Tutorial
Column-Oriented Database Systems
96
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Hazards
l
Data hazards
l l
l
Control Hazards
l l l
Dependencies between instructions L1 data cache misses
Branch mispredictions Computed branches (late binding) L1 instruction cache misses
Result: bubbles in the pipeline
Out-of-order execution addresses data hazards l control hazards typically more expensive
VLDB 2009 Tutorial Column-Oriented Database Systems 97
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
SIMD
l
Single Instruction Multiple Data
l l l
Same operation applied on a vector of values MMX: 64 bits, SSE: 128bits, AVX: 256bits SSE, e.g. multiply 8 short integers
Column-Oriented Database Systems 98
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
102 ivan
37 7
350
PROJECT next()
101 102 alice ivan 37 22 > 30 ? 22 TRUE 37 FALSE
SELECT id, name (age-30)*50 AS bonus FROM employee WHERE age > 30
SELECT next()
102 101 ivan alice 37 22
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
99
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
Operators
Iterator interface -open() -next(): tuple -close()
PROJECT next()
37 22 > 30 ?
SELECT next() SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
100
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
102 ivan
37 7
350
Primitives
Provide computational functionality All arithmetic allowed in expressions, e.g. Multiplication
7 * 50
PROJECT next()
101 102 alice ivan 37 22 > 30 ? 22 TRUE 37 FALSE
SELECT next()
102 101 ivan alice 37 22
SCAN
mult(int,int) Ł int
101
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Database Architecture causes Hazards
l
Data hazards
l l
l
Control Hazards
l l l
Dependencies between instructions L1 data cache misses Out-of-order Execution
SIMD
Branch mispredictions Computed branches (late binding) L1 instruction cache misses
work on one tuple at a time Large Tree/Hash Structures Code footprint of all operators in query plan exceeds L1 cache Data-dependent conditions PROJECT
Operators
Iterator interface -open() -next(): tuple -close()
SELECT
SCAN Next() late binding method calls “DBMSs On A Modern Processor: Where Does Time Go? ” Tree, List, Hash traversal Complex NSM record navigation Ailamaki, DeWitt, Hill, Wood, VLDB’99
VLDB 2009 Tutorial Column-Oriented Database Systems
102
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DBMS Computational Efficiency
TPC-H 1GB, query 1 l selects 98% of fact table, computes net prices and aggregates all l Results:
l l l
C program: MySQL: DBMS “X”:
? 26.2s 28.1s
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
VLDB 2009 Tutorial Column-Oriented Database Systems 103
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DBMS Computational Efficiency
TPC-H 1GB, query 1 l selects 98% of fact table, computes net prices and aggregates all l Results:
l l l
C program: MySQL: DBMS “X”:
0.2s 26.2s 28.1s
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
VLDB 2009 Tutorial Column-Oriented Database Systems 104
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
MonetDB
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a column-store
l
l
“save disk I/O when scan-intensive queries columns” “avoid an expression interpreter to improve computational efficiency”
need a few
VLDB 2009 Tutorial
Column-Oriented Database Systems
106
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
SELECT FROM WHERE
id, name, (age-30)*50 as bonus people age > 30
VLDB 2009 Tutorial
Column-Oriented Database Systems
107
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
CPU happy? Give it “nice” code ! - few dependencies (control,data) - CPU gets out-of-order execution - compiler can e.g. generate SIMD One loop for an entire column batcalc_minus_int(int* res, - no per-tuple interpretation int* col, - arrays: no record navigation int val, - better instruction cache locality
int n) { for(i=0; i<n; i++) as bonus SELECT id, name, (age-30)*50 res[i] = col[i] - val; FROM people } WHERE age > 30
Simple, hardcoded semantics in operators
VLDB 2009 Tutorial
Column-Oriented Database Systems
108
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
CPU happy? Give it “nice” code ! - few dependencies (control,data) - CPU gets out-of-order execution - compiler can e.g. generate SIMD One loop for an entire column batcalc_minus_int(int* res, - no per-tuple interpretation int* col, - arrays: no record navigation int val, - better instruction cache locality
int n) { for(i=0; i<n; i++) as bonus SELECT id, name, (age-30)*50 res[i] = col[i] - val; FROM people } WHERE age > 30
MATERIALIZED intermediate results
VLDB 2009 Tutorial
Column-Oriented Database Systems
109
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a column-store
l l
“save disk I/O when scan-intensive queries
need a few columns”
“avoid an expression interpreter to improve computational efficiency” How?
l
RISC query algebra: hard-coded semantics
l
Decompose complex expressions in multiple operations
l
Operators only handle simple arrays
l
No code that handles slotted buffered record layout
l
Relational algebra becomes array manipulation language
l
Often SIMD for free Plus: use of cache-conscious algorithms for Sort/Aggr/Join
Column-Oriented Database Systems 110
l
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a Faustian pact
l
You want efficiency
l
Simple hard-coded operators Result materialization
l
I take scalability
l
n
C program: MonetDB: MySQL: DBMS “X”:
0.2s 3.7s 26.2s 28.1s
n
n
n
VLDB 2009 Tutorial
Column-Oriented Database Systems
111
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
as a research platform
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
SIGMOD 1985
B MonetD bra lge BAT A ts suppor B MonetD G, L, ODM M SQL, X ..RDF
•“MIL Primitives for Querying a Fragmented World”, Boncz, Kersten, VLDBJ’98 •“Flattening an Object Algebra to Provide Performance” Boncz, Wilschut, Kersten, ICDE’98 •“MonetDB/XQuery: a fast XQuery processor powered C- by a relational engine” Boncz, Grust, vanKeulen, port on e p RDF su SW-Stor Rittinger, Teubner, SIGMOD’06 / •“SW-Store: a vertically partitioned DBMS for Semantic STORE Web data management“ Abadi, Marcus, Madden, Hollenbach, VLDBJ’09
Column-Oriented Database Systems 113
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
The
Extensible query lang frontend framework XQuery .. SGL? Extensible Dynamic
Software Stack
SQL 03 Optimizers MonetDB 5
RDF
Arrays
Runtime QOPT Framework!
SOAP MonetDB 4
MonetDB kernel
Extensible Architecture-Conscious compile Execution platform
OGIS
VLDB 2009 Tutorial
Column-Oriented Database Systems
114
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
optimizer
frontend
backend
as a research platform
l
Cache-Conscious Joins
l
•“Database Architecture Optimized for the New
l
MonetDB/XQuery:
l
Bottleneck: Memory Access” VLDB’99 Cost Models, Radix-cluster, •“Generic Database Cost Models for Hierarchical Memory Radix-decluster Systems”, VLDB’02 (all Manegold, Boncz, Kersten) •“Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
structural joins exploiting positional column access
“MonetDB/XQuery: a fast XQuery processor powered by a relational engine” Boncz, Grust, vanKeulen, Rittinger, Teubner, SIGMOD’06 “Database Cracking”, CIDR’07 “Updating a cracked database “, SIGMOD’07 “Self-organizing tuple reconstruction in columnstores“, SIGMOD’09 (all Idreos, Manegold, Kersten)
l
Cracking:
l
on-the-fly automatic indexing without workload knowledge
l
Recycling:
l
“An architecture for recycling intermediates in a using materialized intermediates column-store”, Ivanova, Kersten, Nes, Goncalves, SIGMOD’09
l
Run-time Query Optimization:
l
correlation-aware run-time optimization without cost model
VLDB 2009 Tutorial
“ROX: run-time optimization of XQueries”, Abdelkader, Boncz, Manegold, vanKeulen, SIGMOD’09
115
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
“MonetDB/X100” vectorized query processing
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB spin-off: MonetDB/X100 Materialization vs Pipelining
VLDB 2009 Tutorial
Column-Oriented Database Systems
117
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
next()
PROJECT next() SELECT next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
118
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
next()
> 30 ?
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
119
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
“Vectorized Observations: Processing”
In Cache
next()
102 ivan 350 104 peggy 750 - 30 * 50 102 ivan 37 7 350 104 peggy 45 15 750 > 30 ?
next() called much less often Ł more time spent in primitives vector = array of ~100 less in overhead processed in a tight loop primitive calls process an array of values cache Resident CPU in a loop:
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
120
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Observations: next() called much less often Ł more time spent in primitives less in overhead primitive calls process an array of values in a loop:
CPU Efficiency depends on “nice” code - out-of-order execution - few dependencies (control,data) - compiler support
102 ivan 350 104 peggy 750 - 30 * 50 102 ivan 37 7 350 104 peggy 45 15 750 > 30 ?
next()
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
Compilers like simple loops over arrays - loop-pipelining - automatic SIMD
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
121
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Observations: next() called much less often Ł more time spent in primitives less in overhead primitive calls process an array of values in a loop:
CPU Efficiency depends on “nice” code - out-of-order execution - few dependencies (control,data) - compiler support Compilers like simple loops over arrays - loop-pipelining - automatic SIMD
> 30 ? FALSE TRUE TRUE FALSE - 30 7 15
for(i=0; i<n; i++) res[i] = (col[i] > x) * 50
350 750
for(i=0; i<n; 30 ? > i++)
PROJECT
FALSE res[i] = (col[i] - x) TRUE TRUE FALSE SELECT for(i=0; i<n; i++)
* 50 350 750
res[i] = (col[i] * x)
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
122
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Tricks being played: - Late materialization - Materialization avoidance using selection vectors
> 30 ?
PROJECT
next()
1 2
SELECT next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
123
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
map_mul_flt_val_flt_col( float *res, int* sel, Tricks being played: float val, float *col, int n) - Late materialization next() { - Materialization avoidance for(int i=0; i<n; i++) using selection vectors res[i] = val * col[sel[i]]; } selection vectors used to reduce vector copying contain selected positions next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
- 30 * 50 7 15 350 750
> 30 ?
PROJECT
next()
1 2
SELECT
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
124
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
map_mul_flt_val_flt_col( float *res, int* sel, Tricks being played: float val, float *col, int n) - Late materialization next() { - Materialization avoidance for(int i=0; i<n; i++) using selection vectors res[i] = val * col[sel[i]]; } selection vectors used to reduce vector copying contain selected positions next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
- 30 * 50 7 15 350 750
> 30 ?
PROJECT
next()
1 2
SELECT
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
125
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB/X100
l
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Both efficiency
l
Vectorized primitives Pipelined query evaluation
l
and scalability..
l
n
C program: MonetDB/X100: MonetDB: MySQL: DBMS “X”:
0.2s 0.6s 3.7s 26.2s 28.1s
Column-Oriented Database Systems 126
n
n
n
n
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory Hierarchy
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 127
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory Hierarchy
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
Vectors are only the in-cache representation RAM & disk representation might actually be different
CPU cache
(vectorwise uses both PAX & DSM)
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 128
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Optimal Vector size?
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
All vectors together should fit the CPU cache Optimizer should tune this, given the query characteristics.
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 129
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
Less and less iterator.next() and primitive function calls (“interpretation overhead”)
VLDB 2009 Tutorial
Column-Oriented Database Systems
130
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
Vectors start to exceed the CPU cache, causing additional memory traffic
VLDB 2009 Tutorial
Column-Oriented Database Systems
131
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
The benefit of selection vectors
VLDB 2009 Tutorial
Column-Oriented Database Systems
132
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
MonetDB/MIL materializes columns
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 133
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Benefits of Vectorized Processing
l
100x less Function Calls
l
iterator.next(), primitives
“Buffering Database Operations for Enhanced Instruction Cache Performance” Zhou, Ross, SIGMOD’04
l
No Instruction Cache Misses
l
l
“Block oriented processing of relational database operations in Less Data Cache Misses modern computer architectures” l Cache-conscious data placement Padmanabhan, Malkemus, Agarwal, ICDE’01 No Tuple Navigation
l
High locality in the primitives
l
Primitives are record-oblivious, only see arrays
l
Vectorization allows algorithmic optimization
l
Move activities out of the loop (“strength reduction”)
l
Compiler-friendly function bodies
l
Loop-pipelining, automatic SIMD
VLDB 2009 Tutorial
Column-Oriented Database Systems
134
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Balancing Vectorized Query Execution with Bandwidth Optimized Storage ” Zukowski, CWI 2008
Vectorizing Relational Operators
l l
Project Select
l
Exploit selectivities, test buffer overflow
l
Aggregation
l
Ordered, Hashed
l
Sort
l
Radix-sort nicely vectorizes
l
Join
l
Merge-join + Hash-join
VLDB 2009 Tutorial
Column-Oriented Database Systems
135
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Efficient Column Store Compression
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Key Ingredients
l
l
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Compress relations on a per-column basis
Columns compress well
l
Decompress small vectors of tuples from a column into the CPU cache
l
Minimize main-memory overhead Exploit processing power of modern CPUs
l
Use light-weight, CPU-efficient algorithms
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
137
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Key Ingredients
l
l
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Compress relations on a per-column basis
Columns compress well
l
Decompress small vectors of tuples from a column into the CPU cache
l
Minimize main-memory overhead
VLDB 2009 Tutorial
Column-Oriented Database Systems
138
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Vectorized Decompression
X100 query engine
Idea: decompress a vector only compression: -between CPU and RAM -Instead of disk and RAM (classic)
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 139
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
RAM-Cache Decompression
l
l
l
l
Decompress vectors on-demand into the cache RAM-Cache boundary only crossed once More (compressed) data cached in RAM Less bandwidth use
VLDB 2009 Tutorial
Column-Oriented Database Systems
140
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Multi-Core Bandwidth & Compression
Performance Degradation with Concurrent Queries
1.2
avg query speed (clock normalized)
1
0.8
0.6
0.4
Intel® Xeon E5410 (Harpertown) - Q6b Normal
0.2
Intel® Xeon E5410 (Harpertown) - Q6b Compressed Intel® Xeon X5560 (Nehalem) - Q6b Normal Intel® Xeon X5560 (Nehalem) - Q6b Compressed
0 1 2 3 4 5 6 7 8
number of concurrent queries
VLDB 2009 Tutorial Column-Oriented Database Systems 141
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
CPU Efficient Decompression
l l
Decoding loop over cache-resident vectors of code words Avoid control dependencies within decoding loop
l
no if-then-else constructs in loop body
l
Avoid data dependencies between loop iterations
VLDB 2009 Tutorial
Column-Oriented Database Systems
142
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Disk Block Layout
l
Forward growing section of arbitrary size code words (code word size fixed per block)
VLDB 2009 Tutorial
Column-Oriented Database Systems
143
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Disk Block Layout
l
l
Forward growing section of arbitrary size code words (code word size fixed per block) Backwards growing exception list
VLDB 2009 Tutorial
Column-Oriented Database Systems
144
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Naïve Decompression Algorithm
l
Use reserved value from code word domain (MAXCODE) to mark exception positions
VLDB 2009 Tutorial
Column-Oriented Database Systems
145
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Deterioriation With Exception%
l
1.2GB/s deteriorates to 0.4GB/s
Column-Oriented Database Systems 146
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Deterioriation With Exception%
l
Perf Counters: CPU mispredicts if-then-else
Column-Oriented Database Systems 147
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patching
l
Maintain a patch-list through code word section that links exception positions
VLDB 2009 Tutorial
Column-Oriented Database Systems
148
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patching
l
Maintain a patch-list through code word section that links exception positions After decoding, patch up the exception positions with the correct values
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
149
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patched Decompression
VLDB 2009 Tutorial
Column-Oriented Database Systems
150
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patched Decompression
VLDB 2009 Tutorial
Column-Oriented Database Systems
151
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Decompression Bandwidth
Patching can be applied to: • Frame of Reference (PFOR) • Delta Compression (PFOR-DELTA) • Dictionary Compression (PDICT)
Makes these methods much more applicable to noisy data Ł without additional cost
l
Patching makes two passes, but is faster!
Column-Oriented Database Systems 152
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Conclusion
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary (1/2)
l
Columns and Row-Stores: different?
l l
No fundamental differences Can current row-stores simulate column-stores now?
l
not efficiently: row-stores need change actually independent issues, on-the-fly conversion pays off column favors sequential access, row random Fractured mirrors PAX, Clotho Data morphing
l
On disk layout vs execution layout
l l
l
Mixed Layout schemes
l l l
VLDB 2009 Tutorial
Column-Oriented Database Systems
154
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary (2/2)
l
Crucial Columnar Techniques
l
Storage
l
Lean headers, sparse indices, fast positional access Operating on compressed data Lightweight, vectorized decompression Non-join: LM always wins Naïve/Invisible/Jive/Flash/Radix Join (LM often wins) Vectorized in-cache execution Exploiting SIMD
Column-Oriented Database Systems 155
l
Compression
l l
l
Late vs Early materialization
l l
l
Execution
l l
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Future Work
l
looking at write/load tradeoffs in column-stores
l
read-only vs batch loads vs trickle updates vs OLTP
VLDB 2009 Tutorial
Column-Oriented Database Systems
156
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (1/3)
l
Column-stores are update-in-place averse
l l l l
In-place: I/O for each column + re-compression + multiple sorted replicas + sparse tree indices
Update-in-place is infeasible!
VLDB 2009 Tutorial
Column-Oriented Database Systems
157
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (2/3)
l
Column-stores use differential mechanisms instead
l l l
Differential lists/files or more advanced (e.g. PDTs) Updates buffered in RAM, merged on each query Checkpointing merges differences in bulk sequentially
l
I/O trends favor this anyway
§ §
trade RAM for converting random into sequential I/O this trade is also needed in Flash (do not write randomly!) Depends on available RAM for buffering (how long until full) § Checkpoint must be done within that time § The longer it can run, the less it molests queries Using Flash for buffering differences buys a lot of time § Hundreds of GBs of differences per server
l
How high loads can it sustain?
§
§
VLDB 2009 Tutorial
Column-Oriented Database Systems
158
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (3/3)
l l
Differential transactions favored by hardware trends Snapshot semantics accepted by the user community
l
can always convert to serialized
“Serializable Isolation For Snapshot Databases” Alomari, Cahill, Fekete, Roehm, SIGMOD’08
Ł
Row stores could also use differential transactions and be efficient!
Ł Ł
Implies a departure from ARIES Implies a full rewrite
My conclusion: a system that combines row- and columns needs differentially implemented transactions. Starting from a pure column-store, this is a limited add-on. Starting from a pure row-store, this implies a full rewrite.
VLDB 2009 Tutorial Column-Oriented Database Systems 159
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Future Work
l
looking at write/load tradeoffs in column-stores
l
read-only vs batch loads vs trickle updates vs OLTP
l l
database design for column-stores column-store specific optimizers
l
compression/materialization/join tricks Ł cost models? can row-stores learn new column tricks?
l
l
hybrid column-row systems
l
l
Study of the minimal number changes one needs to make to a row store to get the majority of the benefits of a column-store Alternative: add features to column-stores that make them more like row stores
VLDB 2009 Tutorial
Column-Oriented Database Systems
160
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Conclusion
l
Columnar techniques provide clear benefits for:
l l
Data warehousing, BI Information retrieval, graphs, e-science Without these, existing row systems do not benefit Row-stores may adopt some column-store techniques Column-stores add row-store (or PAX) functionality
l
A number of crucial techniques make them effective
l
l
Row-Stores and column-stores could be combined
l l
l
Many open issues to do research on!
VLDB 2009 Tutorial
Column-Oriented Database Systems
161
Column-Oriented Database Systems
Part 1: Stavros Harizopoulos (HP Labs) Part 2: Daniel Abadi (Yale) Part 3: Peter Boncz (CWI)
VLDB 2009 Tutorial
VLDB 2009 Tutorial Column-Oriented Database Systems
1
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What is a column-store?
row-store
Date Store Product Customer Price
column-store
Date Store Product Customer Price
+ easy to add/modify a record - might read in unnecessary data
+ only need to read in relevant data - tuple writes require multiple accesses
=> suitable for read-mostly, read-intensive, large data repositories
VLDB 2009 Tutorial
Column-Oriented Database Systems
2
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Are these two fundamentally different?
l l
The only fundamental difference is the storage layout However: we need to look at the big picture
different storage layouts proposed row-stores row-stores++ ‘90s ‘00s row-stores++ today column-stores converge?
‘70s
‘80s
new applications new bottleneck in hardware
l l l
How did we get here, and where we are heading Part 1 What are the column-specific optimizations? Part 2 How do we improve CPU efficiency when operating on Cs
Part 3
VLDB 2009 Tutorial Column-Oriented Database Systems 3
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
4
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco Data Warehousing example
l
Typical DW installation Real-world example
dimension tables account or RAM usage toll source fact table
l
“One Size Fits All? - Part 2: Benchmarking Results” Stonebraker et al. CIDR 2007
QUERY 2 SELECT account.account_number, sum (usage.toll_airtime), sum (usage.toll_price) FROM usage, toll, source, account WHERE usage.toll_id = toll.toll_id AND usage.source_id = source.source_id AND usage.account_id = account.account_id AND toll.type_ind in (‘AE’. ‘AA’) AND usage.toll_price > 0 AND source.type != ‘CIBER’ AND toll.rating_method = ‘IS’ AND usage.invoice_date = 20051013 GROUP BY account.account_number
star schema
Query 1 Query 2 Query 3 Query 4 Query 5
Column-store 2.06 2.20 0.09 5.24 2.88
Row-store 300 300 300 300 300
Why? Three main factors (next slides)
Column-Oriented Database Systems 5
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (1/3): read efficiency
row store column store
read pages containing entire rows one row = 212 columns! is this typical? (it depends)
What about vertical partitioning? (it does not work with ad-hoc queries)
VLDB 2009 Tutorial
read only columns needed in this example: 7 columns caveats: • “select * ” not any faster • clever disk prefetching • clever tuple reconstruction
Column-Oriented Database Systems
6
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (2/3): compression efficiency
l
Columns compress better than rows
l l
Typical row-store compression ratio 1 : 3 Column-store 1 : 10
l
Why?
l
l l
Rows contain values from different domains => more entropy, difficult to dense-pack Columns exhibit significantly less entropy Male, Female, Female, Female, Male Examples:
1998, 1998, 1999, 1999, 1999, 2000
l
Caveat: CPU cost (use lightweight compression)
VLDB 2009 Tutorial
Column-Oriented Database Systems
7
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Telco example explained (3/3): sorting & indexing efficiency
l
Compression and dense-packing free up space
l l l l
Use multiple overlapping column collections Sorted columns compress better Range queries are faster Use sparse clustered indexes
What about heavily-indexed row-stores? (works well for single column access, cross-column joins become increasingly expensive)
VLDB 2009 Tutorial
Column-Oriented Database Systems
8
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Additional opportunities for column-stores
l
Block-tuple / vectorized processing
l
Easier to build block-tuple operators
l
Amortizes function-call cost, improves CPU cache performance Software-based: bitwise operations Hardware-based: SIMD Part 3
l
Easier to apply vectorized primitives
l l
l
Opportunities with compressed columns
l l
Avoid decompression: operate directly on compressed Delay decompression (and tuple reconstruction)
l
Also known as: late materialization
more in Part 2
l
Exploit columnar storage in other DBMS components
l
Physical design (both static and dynamic)
VLDB 2009 Tutorial Column-Oriented Database Systems
See: Database Cracking, from CWI
9
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Effect on C-Store performance
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Hachem, and Madden. SIGMOD 2008.
Average for SSBM queries on C-store
Time (sec)
original C-store
column-oriented join algorithm
enable compression & operate on compressed
enable late materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
10
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary of column-store key features
columnar storage header/ID elimination l Part 1 Part 3
Storage layout
compression
Part 2
multiple sort orders
column operators
Part 1
Part 2
l
Execution engine
avoid decompression late materialization Part 2 Part 3
vectorized operations l
Design tools, optimizer
VLDB 2009 Tutorial Column-Oriented Database Systems 11
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
12
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
From DSM to Column-stores
70s -1985:
TOD: Time Oriented Database – Wiederhold et al. "A Modular, Self-Describing Clinical Databank System," Computers and Biomedical Research, 1975 More 1970s: Transposed files, Lorie, Batory, Svensson. “An overview of cantor: a new system for data analysis” Karasalo, Svensson, SSDBM 1983 “A decomposition storage model” Copeland and Khoshafian. SIGMOD 1985.
1985: DSM paper
1990s: Commercialization through SybaseIQ Late 90s – 2000s: Focus on main-memory performance
l l
DSM “on steroids” [1997 – now] Hybrid DSM/NSM [2001 – 2004]
CWI: MonetDB Wisconsin: PAX, Fractured Mirrors CMU: Clotho
Michigan: Data Morphing MIT: C-Store
VLDB 2009 Tutorial
2005 – : Re-birth of read-optimized DSM as “column-store”
CWI: MonetDB/X100
Column-Oriented Database Systems
10+ startups
13
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
The original DSM paper
l l l l
“A decomposition storage model” Copeland and Khoshafian. SIGMOD 1985.
Proposed as an alternative to NSM 2 indexes: clustered on ID, non-clustered on value Speeds up queries projecting few columns Requires more storage value
ID
0100 0962 1000 .. 1 2 3 4 ..
VLDB 2009 Tutorial
Column-Oriented Database Systems
14
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory wall and PAX
l
90s: Cache-conscious research
“Cache Conscious Algorithms for from: Relational Query Processing.” Shatdal, Kant, Naughton. VLDB 1994. “Database Architecture Optimized for to: the New Bottleneck: Memory Access.” Boncz, Manegold, Kersten. VLDB 1999.
“DBMSs on a modern processor: and: Where does time go?” Ailamaki, DeWitt, Hill, Wood. VLDB 1999.
l
PAX: Partition Attributes Across
l l
Retains NSM I/O pattern Optimizes cache-to-RAM communication
“Weaving Relations for Cache Performance.” Ailamaki, DeWitt, Hill, Skounakis, VLDB 2001.
VLDB 2009 Tutorial
Column-Oriented Database Systems
15
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
More hybrid NSM/DSM schemes
l
Dynamic PAX: Data Morphing
“Data morphing: an adaptive, cache-conscious storage technique.” Hankins, Patel, VLDB 2003.
l
Clotho: custom layout using scatter-gather I/O
“Clotho: Decoupling Memory Page Layout from Storage Organization.” Shao, Schindler, Schlosser, Ailamaki, and Ganger. VLDB 2004.
l
Fractured mirrors
l
Smart mirroring with both NSM/DSM copies
“A Case For Fractured Mirrors.” Ramamurthy, DeWitt, Su, VLDB 2002.
VLDB 2009 Tutorial
Column-Oriented Database Systems
16
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB (more in Part 3)
l l
Late 1990s, CWI: Boncz, Manegold, and Kersten Motivation:
l l
l l
Main-memory Improve computational efficiency by avoiding expression interpreter DSM with virtual IDs natural choice Developed new query execution algebra Pointed out memory-wall in DBMSs Cache-conscious projections and joins …
VLDB 2009 Tutorial Column-Oriented Database Systems 17
l
Initial contributions:
l l l
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
2005: the (re)birth of column-stores
l
New hardware and application realities
l l
Faster CPUs, larger memories, disk bandwidth limit Multi-terabyte Data Warehouses Read-optimized, fast multi-column access, disk/CPU efficiency, light-weight compression
l
New approach: combine several techniques
l
l
C-store paper:
l
First comprehensive design description of a column-store “proper” disk-based column store
l
MonetDB/X100
l
l
Explosion of new products
VLDB 2009 Tutorial Column-Oriented Database Systems 18
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Performance tradeoffs: columns vs. rows
DSM traditionally was not favored by technology trends How has this changed?
l l
Optimized DSM in “Fractured Mirrors,” 2002 “Apples-to-apples” comparison “Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
l l
Follow-up study
“Read-Optimized Databases, InDepth” Holloway, DeWitt, VLDB’08
Main-memory DSM vs. NSM
“DSM vs. NSM: CPU performance tradeoffs in block-oriented query processing” Boncz, Zukowski, Nes, DaMoN’08
l
Flash-disks: a come-back for PAX?
“Query Processing Techniques “Fast Scans and Joins Using Flash for Solid State Drives” Drives” Shah, Harizopoulos, Tsirogiannis, Harizopoulos, Wiener, Graefe. DaMoN’08 Shah, Wiener, Graefe, VLDB 2009 Tutorial Column-Oriented Database Systems 19 SIGMOD’09
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Fractured mirrors: a closer look
l
Store DSM relations inside a B-tree
l l l
Tuple Header
Leaf nodes contain values Eliminate IDs, amortize header overhead Custom implementation on Shore
TID Column Data
“A Case For Fractured Mirrors” Ramamurthy, DeWitt, Su, VLDB 2002.
sparse B-tree on ID 3
1 2 3 4 5
a1 a2 a3 a4 a5
1
a1 1
2 a1
a2
3
a3
4 4
a4
5
a5
a2 a3
a4 a5
Similar: storage density “Efficient columnar comparable storage in B-trees” Graefe. to column stores Sigmod Record 03/2007.
VLDB 2009 Tutorial Column-Oriented Database Systems 20
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Fractured mirrors: performance
From PAX paper: time regular DSM column? column? row
columns projected: 1 2 3 4 5
l
Chunk-based tuple merging
l l l
optimized DSM
Read in segments of M pages Merge segments in memory Becomes CPU-bound after 5 pages
VLDB 2009 Tutorial Column-Oriented Database Systems 21
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-scanner implementation
row scanner
“Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
column scanner
Joe 45 … …
Joe 45 … …
SELECT name, age WHERE age > 40
apply predicate(s)
S
Direct I/O Joe Sue … prefetch ~100ms worth of data
S
#POS 45 #POS …
1 Joe 45 2 Sue 37 …… …
apply predicate #1 45 37 …
S
VLDB 2009 Tutorial
Column-Oriented Database Systems
22
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Scan performance
l l l l l
Large prefetch hides disk seeks in columns Column-CPU efficiency with lower selectivity Row-CPU suffers from memory stalls Memory stalls disappear in narrow tuples Compression: similar to narrow
not shown, details in the paper
VLDB 2009 Tutorial
Column-Oriented Database Systems
23
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Even more results
• •
“Read-Optimized Databases, InDepth” Holloway, DeWitt, VLDB’08
35
Same engine as before Additional findings
narrow & compressed tuple: 30 CPU-bound!
25
C-25% C-10% R-50%
Time (s)
20
15
10
wide attributes: same as before
5
0 1 2 3 4 5 6 7 Columns Returned 8 9 10
Non-selective queries, narrow tuples, favor well-compressed rows Materialized views are a win Column-joins are Scan times determine early materialized joins covered in part 2!
VLDB 2009 Tutorial Column-Oriented Database Systems 24
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Speedup of columns over rows
cycles per disk byte
144 72 36 18 9 “Performance Tradeoffs in ReadOptimized Databases” Harizopoulos, Liang, Abadi, Madden, VLDB’06
(cpdb)
+++
_ = + ++
8 12 16 20 24 28 32 36
tuple width
l
Rows favored by narrow tuples and low cpdb
l
Disk-bound workloads have higher cpdb
VLDB 2009 Tutorial Column-Oriented Database Systems 25
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Varying prefetch size
no competing disk traffic
40
time (sec)
Column 2
30 20 10 0 4 8 12 16 20 24 28 32
Column 8 Column 16 Column 48 (x 128KB) Row (any prefetch size)
selected bytes per tuple
l
No prefetching hurts columns in single scans
VLDB 2009 Tutorial Column-Oriented Database Systems 26
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Varying prefetch size
with competing disk traffic
40
time (sec)
30 20 10 0 4
Column, 48 Row, 48
40 30 20 10 0 Column, 8 Row, 8 4 12 20 28
12
20
28
selected bytes per tuple
l l
No prefetching hurts columns in single scans Under competing traffic, columns outperform rows for any prefetch size
VLDB 2009 Tutorial Column-Oriented Database Systems 27
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Performance
l l l
“DSM vs. NSM: CPU performance trade offs in block-oriented query processing” Boncz, Zukowski, Nes, DaMoN’08
Benefit in on-the-fly conversion between NSM and DSM DSM: sequential access (block fits in L2), random in L1 NSM: random access, SIMD for grouped Aggregation
VLDB 2009 Tutorial
Column-Oriented Database Systems
28
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
New storage technology: Flash SSDs
l
Performance characteristics
l l
very fast random reads, slow random writes fast sequential reads and writes
l
Price per bit (capacity follows)
l
cheaper than RAM, order of magnitude more expensive than Disk
avoid random writes! SSD (Ł small reads still suffer from SATA overhead/OS limitations) PCI card (Ł high price, limited expandability)
l
Flash Translation Layer introduces unpredictability
l
l
Form factors not ideal yet
l l
l
Boost Sequential I/O in a simple package
l
Flash RAID: very tight bandwidth/cm3 packing (4GB/sec inside the box) useful for delta structures and logs still suboptimal if I/O block size > record size therefore column stores profit mush less than horizontal stores the larger the data, the deeper clustering one can exploit
Column-Oriented Database Systems 29
l
Column Store Updates
l
l
Random I/O on flash fixes unclustered index access
l l
l
Random I/O useful to exploit secondary, tertiary table orderings
l VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Even faster column scans on flash SSDs
l
New-generation SSDs
l l
30K Read IOps, 3K Write Iops 250MB/s Read BW, 200MB/s Write
Very fast random reads, slower random writes Fast sequential RW, comparable to HDD arrays
l l
No expensive seeks across columns FlashScan and Flashjoin: PAX on SSDs, inside Postgres
“Query Processing Techniques for Solid State Drives” Tsirogiannis, Harizopoulos, Shah, Wiener, Graefe, SIGMOD’09 mini-pages with no qualified attributes are not accessed
VLDB 2009 Tutorial
Column-Oriented Database Systems
30
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-scan performance over time
regular DSM (2001) from 7x slower column-store (2006) ..to 1.2x slower
..to 2x slower
..to same
and 3x faster! optimized DSM (2002)
VLDB 2009 Tutorial
SSD Postgres/PAX (2009)
Column-Oriented Database Systems 31
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
32
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Architecture of a column-store
storage layout
l l l l
read-optimized: dense-packed, compressed organize in extends, batch updates multiple sort orders engine sparse indexes l block-tuple operators l new access methods l optimized relational operators system-level system-wide column support loading / updates scaling through multiple nodes transactions / redundancy
VLDB 2009 Tutorial Column-Oriented Database Systems 33
l l l l
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
C-Store
l l l l l l l l l l l
“C-Store: A Column-Oriented DBMS.” Stonebraker et al. VLDB 2005.
Compress columns No alignment Big disk blocks Only materialized views (perhaps many) Focus on Sorting not indexing Data ordered on anything, not just time Automatic physical DBMS design Optimize for grid computing Innovative redundancy Xacts – but no need for Mohan Column optimizer and executor
VLDB 2009 Tutorial Column-Oriented Database Systems 34
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
C-Store: only materialized views (MVs)
l l
l l l l l
Projection (MV) is some number of columns from a fact table Plus columns in a dimension table – with a 1-n join between Fact and Dimension table Stored in order of a storage key(s) Several may be stored! With a permutation, if necessary, to map between them Table (as the user specified it and sees it) is not stored! No secondary indexes (they are a one column sorted MV plus a permutation, if you really want one)
User view:
EMP (name, age, salary, dept) Dept (dname, floor)
Possible set of MVs:
MV-1 (name, dept, floor) in floor order MV-2 (salary, age) in age order MV-3 (dname, salary, name) in salary order
Column-Oriented Database Systems 35
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Continuous Load and Query (Vertica)
Hybrid Storage Architecture
> Write Optimized Store (WOS)
> Read Optimized Store (ROS)
TUPLE MOVER • On disk • Sorted / Compressed • Segmented • Large data loaded direct
Trickle Load
A B C
Asynchronous Data Transfer
§Memory based §Unsorted / Uncompressed §Segmented §Low latency / Small quick inserts
VLDB 2009 Tutorial Column-Oriented Database Systems
A
B
C
(A B C | A)
36
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Loading Data (Vertica)
> INSERT, UPDATE, DELETE > Bulk and Trickle Loads
Write-Optimized Store (WOS) In-memory Automatic Tuple Mover
§COPY §COPY DIRECT
> User loads data into logical Tables > Vertica loads atomically into storage
Read-Optimized Store (ROS) On-disk
VLDB 2009 Tutorial
Column-Oriented Database Systems
37
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Applications for column-stores
l
Data Warehousing
l l l
High end (clustering) Mid end/Mass Market Personal Analytics E.g. Proximity
l
Data Mining
l
l l
Google BigTable RDF
l
Semantic web data management Terabyte TREC SciDB initiative SLOAN Digital Sky Survey on MonetDB
l
Information retrieval
l
l
Scientific datasets
l l
VLDB 2009 Tutorial
Column-Oriented Database Systems
38
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
List of column-store systems
l l l l l l l l l l l l l
Cantor (history) Sybase IQ SenSage (former Addamark Technologies) Kdb 1010data MonetDB C-Store/Vertica X100/VectorWise KickFire SAP Business Accelerator Infobright ParAccel Exasol
VLDB 2009 Tutorial Column-Oriented Database Systems 39
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
Part 1: Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
40
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Simulate a Column-Store inside a Row-Store
Date Store Product Customer Price
01/01 BOS Table 01/01 NYC Chair 01/01 BOS Bed
Mesa Lutz Mudd
$20 $13 $79
Option B: Index Every Column
Date Index
Option A: Vertical Partitioning
Date
TID Value
Store
TID Value
Product
TID Value
Customer
TID Value TID
Price
Value
1 01/01 2 01/01 3 01/01
1 2 3
BOS NYC BOS
1 Table 2 Chair 3 Bed
1 2 3
Mesa Lutz Mudd
1 2 3
$20 $13 $79
Store Index
…
VLDB 2009 Tutorial Column-Oriented Database Systems 41
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Simulate a Column-Store inside a Row-Store
Date Store Product Customer Price
01/01 BOS Table 01/01 NYC Chair 01/01 BOS Bed
Mesa Lutz Mudd
$20 $13 $79
Option B: Index Every Column
Date Index
Option A: Vertical Partitioning
Date
Value StartPos Length
Store
TID Value
Product
TID Value
Customer
TID Value TID
Price
Value
01/01
1
3
Can explicitly runlength encode date
1 2 3
BOS NYC BOS
1 Table 1 2 Chair 2 3 Bed 3
Mesa Lutz Mudd
1 2 3
$20 $13 $79
Store Index
“Teaching an Old Elephant New Tricks.” Bruno, CIDR 2009.
VLDB 2009 Tutorial Column-Oriented Database Systems
…
42
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Experiments
l
Star Schema Benchmark (SSBM)
Adjoined Dimension Column Index (ADC Index) to Improve Star Schema Query Performance”. O’Neil et. al. ICDE 2008.
l l
Implemented by professional DBA Original row-store plus 2 column-store simulations on same row-store product
250.0 200.0
Time (seconds)
150.0 100.0 50.0 0.0
Normal Row-Store Average 25.7
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Hachem, and Madden. SIGMOD 2008.
Vertically Partitioned Row-Store 79.9
Row-Store With All Indexes 221.2
VLDB 2009 Tutorial
Column-Oriented Database Systems
43
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What’s Going On? Vertical Partitions
l
Vertical partitions in row-stores:
l l l
Work well when workload is known ..and queries access disjoint sets of columns See automated physical design
Tuple Header TID Column Data
l
Do not work well as full-columns
l l
TupleID overhead significant Excessive joins
1 2 3
Queries touch 3-4 foreign keys in fact table, 1-2 numeric columns
“Column-Stores vs. Row-Stores: How Different Are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
VLDB 2009 Tutorial
Complete fact table takes up ~4 GB (compressed) Vertically partitioned tables take up 0.7-1.1 GB (compressed)
Column-Oriented Database Systems 44
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What’s Going On? All Indexes Case
l
Tuple construction
l
Common type of query:
SELECT store_name, SUM(revenue) FROM Facts, Stores WHERE fact.store_id = stores.store_id AND stores.country = “Canada” GROUP BY store_name
l
l
Result of lower part of query plan is a set of TIDs that passed all predicates Need to extract SELECT attributes at these TIDs
l l
BUT: index maps value to TID You really want to map TID to value (i.e., a vertical partition)
Tuple construction is SLOW
VLDB 2009 Tutorial Column-Oriented Database Systems 45
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
So….
l l
All indexes approach is a poor way to simulate a column-store Problems with vertical partitioning are NOT fundamental
l l l
Store tuple header in a separate partition Allow virtual TIDs Combine clustered indexes, vertical partitioning Might be possible, BUT: l Need better support for vertical partitioning at the storage layer l Need support for column-specific optimizations at the executer level l Full integration: buffer pool, transaction manager, ..
l
So can row-stores simulate column-stores?
l
l
When will this happen?
l
Most promising features = soon
See Part 2, Part 3 for most promising features
l
..unless new technology / new objectives change the game (SSDs, Massively Parallel Platforms, Energy-efficiency)
Column-Oriented Database Systems 46
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
End of Part 1
l
Basic concepts — Stavros
l l l l
Introduction to key features From DSM to column-stores and performance tradeoffs Column-store architecture overview Will rows and columns ever converge?
l
Part 2: Column-oriented execution — Daniel Part 3: MonetDB/X100 and CPU efficiency — Peter
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
47
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Part 2 Outline
l
Compression Tuple Materialization Joins
l
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
48
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Compression
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06 “Integrating Compression and Execution in ColumnOriented Database Systems” Abadi, Madden, and Ferreira, SIGMOD ’06 •Query optimization in compressed database systems” Chen, Gehrke, Korn, SIGMOD’01
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Compression
l l
Trades I/O for CPU Increased column-store opportunities:
l l
l
Higher data value locality in column stores Techniques such as run length encoding far more useful Can use extra space to store multiple copies of data in different sort orders
VLDB 2009 Tutorial
Column-Oriented Database Systems
50
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Run-length Encoding
Quarter Product ID Price Q1 Q1 Q1 Q1 Q1 Q1 Q1 … Q2 Q2 Q2 Q2 … 1 1 1 1 1 2 2 … 1 1 1 2 … 5 7 2 9 6 8 5 … 3 8 1 4 …
Column-Oriented Database Systems
Quarter
(value, start_pos, run_length)
Product ID (1, 1, 5) (2, 6, 2) …
Price 5 7 2 9 6 8 5 … 3 8 1 4 …
51
(value, start_pos, run_length)
(Q1, 1, 300) (Q2, 301, 350) (Q3, 651, 500) (Q4, 1151, 600)
(1, 301, 3) (2, 304, 1) …
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Bit-vector Encoding
l
For each unique Product ID value, v, in column 1 c, create bit-vector 1 b
l
ID: 1 1 1 1 1 1 0 0 … 1 1 0 0 …
ID: 2 0 0 0 0 0 1 1 … 0 0 1 0 …
ID: 3 0 0 0 0 0 0 0 … 0 0 0 1 …
… 0 0 0 0 0 0 0 … 0 0 0 0 …
52
b[i] = 1 if c[i] = v
l
l
Good for columns with few unique values Each bit-vector can be further compressed if sparse
1 1 1 2 2 … 1 1 2 3 …
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Dictionary Encoding
l
l
l
For each unique value create dictionary entry Dictionary can be per-block or per-column Column-stores have the advantage that dictionary entries may encode multiple values at once
Quarter Quarter Q1 Q2 Q4 Q1 Q3 Q1 Q1 Q1 Q2 Q4 Q3 Q3 …
0 1 3 0 2 0 0 0 1 3 2 2
Quarter 24 128 122
+
Dictionary 0: Q1 1: Q2 2: Q3 3: Q4
OR
+
Dictionary 24: Q1, Q2, Q4, Q1 … 122: Q2, Q4, Q3, Q3 … 128: Q3, Q1, Q1, Q1
53
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Frame Of Reference Encoding
l
l
Encodes values as b bit offset from chosen frame of reference Special escape code (e.g. all bits set to 1) indicates a difference larger than can be stored in b bits
l
Price 45 54 48 55 51 53 40 50 49 62 52 50 …
Price
Frame: 50
After escape code, original (uncompressed) value is written
“Compressing Relations and Indexes ” Goldstein, Ramakrishnan, Shaft, ICDE’98
VLDB 2009 Tutorial
-5 4 -2 5 1 3 ∞ 40 0 -1 ∞ 62 2 0 …
4 bits per value
Exceptions (see part 3 for a better way to deal with exceptions)
Column-Oriented Database Systems
54
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Differential Encoding
l l
Encodes values as b bit offset from previous value Special escape code (just like frame of reference encoding) indicates a difference larger than can be stored in b bits
l
Time 5:00 5:02 5:03 5:03 5:04 5:06 5:07 5:08 5:10 5:15 5:16 5:16 …
Time 5:00 2 1 0 1 2 1 1 2 ∞ 5:15 1 0
After escape code, original (uncompressed) value is written
2 bits per value
l
Performs well on columns containing increasing/decreasing sequences
l l l l
inverted lists timestamps object IDs sorted / clustered columns
Exception (see part 3 for a better way to deal with exceptions)
“Improved Word-Aligned Binary Compression for Text Indexing” Ahn, Moffat, TKDE’06
VLDB 2009 Tutorial
Column-Oriented Database Systems
55
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What Compression Scheme To Use?
Does column appear in the sort key? yes Is the average run-length > 2 yes RLE no Differential Encoding yes no Does this column appear frequently in selection predicates? no Are number of unique values < ~50000
Is the data numerical and exhibit good locality? yes Frame of Reference Encoding no Leave Data Uncompressed
yes
no
Bit-vector Compression
Dictionary Compression
OR
Heavyweight Compression
VLDB 2009 Tutorial
Column-Oriented Database Systems
56
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Heavy-Weight Compression Schemes
l l
Ł Ł
Modern disk arrays can achieve > 1GB/s 1/3 CPU for decompression Ł 3GB/s needed
Lightweight compression schemes are better Even better: operate directly on compressed data
VLDB 2009 Tutorial Column-Oriented Database Systems 57
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
l l l
I/O - CPU tradeoff is no longer a tradeoff Reduces memory–CPU bandwidth requirements Opens up possibility of operating on multiple records at once
VLDB 2009 Tutorial
Column-Oriented Database Systems
58
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
Quarter Product ID 1 … (Q1, 1, 300) (Q2, 301, 6) (Q3, 307, 500) (Q4, 807, 600)
301-306
2 … 0 0 1 0 0 0 1 0 0 1 0 …
3 … 0 1 0 0 0 1 0 0 1 0 1 …
ProductID, COUNT(*))
Index Lookup + Offset jump
1 0 0 1 1 0 0 1 0 0 0 …
(1, 3) (2, 1) (3, 2) SELECT ProductID, Count(*) FROM table WHERE (Quarter = Q2) GROUP BY ProductID
VLDB 2009 Tutorial
Column-Oriented Database Systems
59
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Integrating Compression and Execution in ColumnOriented Database Systems” Abadi et. al, SIGMOD ’06
Operating Directly on Compressed Data
Block API Data Aggregation Operator isOneValue() isValueSorted() isPosContiguous() isSparse() getNext() decompressIntoArray() getValueAtPosition(pos) getMin() getMax() getSize() (Q1, 1, 300) (Q2, 301, 6) (Q3, 307, 500) (Q4, 807, 600)
SELECT ProductID, Count(*) FROM table WHERE (Quarter = Q2) GROUP BY ProductID
Selection Operator
CompressionAware Scan Operator
VLDB 2009 Tutorial
Column-Oriented Database Systems
60
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Tuple Materialization and Column-Oriented Join Algorithms
“Materialization Strategies in a ColumnOriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007. “Self-organizing tuple reconstruction in column-stores“, Idreos, Manegold, Kersten, SIGMOD’09 “Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008. “Query Processing Techniques for Solid State Drives” Tsirogiannis, Harizopoulos Shah, Wiener, and Graefe. SIGMOD 2009. “Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
When should columns be projected?
l
Where should column projection operators be placed in a query plan?
l
Row-store:
l l
Column projection involves removing unneeded columns from tuples Generally done as early as possible Operation is almost completely opposite from a row-store Column projection involves reading needed columns from storage and extracting values for a listed set of tuples
§
l
Column-store:
l l
This process is called “materialization”
Straightforward since there is a one-to-one mapping across columns More complicated since selection and join operators on one column obfuscates mapping to other columns from same table Many database interfaces expect output in regular tuples (rows) Rest of discussion will focus on this case
l l
Early materialization: project columns at beginning of query plan
§
Late materialization: wait as long as possible for projecting columns
§
l
Most column-stores construct tuples and column projection time
§ §
VLDB 2009 Tutorial
Column-Oriented Database Systems
62
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
When should tuples be constructed?
Select + Aggregate 4 4 4 4 2 1 3 1 2 7 3 13 3 42 3 80
l
QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
Construct (4,1,4) 2 1 3 1
storeID
Solution 1: Create rows first (EM). But:
l l l
2 3 3 3
7 13 42 80
Need to construct ALL tuples Need to decompress data Poor memory bandwidth utilization
prodID
custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
63
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
1 1 1 1
Data Source price Data Source Data Source
AGG
Data Source custID
0 1 0 1
QUERY: SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND 4 4 4 4
prodID
Data Source prodID
Data Source storeID
2 1 3 1
storeID
4 4 4 4
2 1 3 1
2 3 3 3
7 13 42 80
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
64
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG
Data Source custID Data Source price
0 1 0 1
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND AND 1 1 1 1 0 1 0 1 4 4 4 4 2 1 3 1 2 3 3 3 7 13 42 80
Data Source prodID
Data Source storeID
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
65
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG
Data Source custID Data Source price
3 3 2 0 3 1 3 0 3 1
custID
13 1 80 1 7 0 13 1 42 0 80 1
price
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
Data Source
Data Source
AND
Data Source prodID
Data Source storeID
0 1 0 1
4 4 4 4
2 1 3 1
2 3 3 3
7 13 42 80
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
66
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Solution 2: Operate on columns
QUERY:
AGG 3 1 1 93
Data Source custID Data Source price
SELECT custID,SUM(price) FROM table WHERE (prodID = 4) AND (storeID = 1) AND GROUP BY custID
AND
AGG 4 4 4 4 2 1 3 1 2 3 3 3 7 13 42 80
Data Source prodID
Data Source storeID
3 1 3 1
13 1 80 1
prodID storeID custID price
VLDB 2009 Tutorial
Column-Oriented Database Systems
67
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Materialization Strategies in a Column-Oriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007.
For plans without joins, late materialization is a win
10 9 8 7 6 5 4 3 2 1 0 Low selectivity Medium selectivity High selectivity
QUERY: SELECT C1, SUM(C2) FROM table WHERE (C1 < CONST) AND (C2 < CONST) GROUP BY C1
l
Time (seconds)
Ran on 2 compressed columns from TPC-H scale 10 data
Early Materialization
Late Materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
68
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Materialization Strategies in a Column-Oriented DBMS” Abadi, Myers, DeWitt, and Madden. ICDE 2007.
Even on uncompressed data, late materialization is still a win
14 12
QUERY: SELECT C1, SUM(C2) FROM table WHERE (C1 < CONST) AND (C2 < CONST) GROUP BY C1
Time (seconds)
10 8 6 4 2 0 Low selectivity Medium selectivity High selectivity
l
Materializing late still works best
Early Materialization
Late Materialization
VLDB 2009 Tutorial
Column-Oriented Database Systems
69
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What about for plans with joins?
Select R1.B, R1.C, R2.E, R2.H, R3.F From R1, R2, R3 Where R1.A = R2.D AND R2.G = R3.K B, C, E, H, F B, C
Join 2 G=K
B, C, E, H, F
Project Join G=K
F K
Scan
G F, K
Project Join A=D
B, C, E, G, H
Join 1 A=D
R3 (F, K, L) Scan R3 (F, K, L)
G D
Scan
E, H
A, B, C
Scan
D, E, G, H
Scan
A
Scan
Column-Oriented Database Systems
R1 (A, B, C) R2 (D, E, G, H)
VLDB 2009 Tutorial
R1 (A, B, C) R2 (D, E, G, H)
70
70
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
What about for plans with joins?
Select R1.B, R1.C, R2.E, R2.H, R3.F From R1, R2, R3 Where R1.A = R2.D AND R2.G = R3.K B, C, E, H, F B, C
Join 2 G=K
B, C, E, H, F
Project Join G=K
F K
Scan
G F, K
B, C, E, G, H Early
Late Project
Join A=D
materialization Join 1
A=D Scan R3 (F, K, L)
materialization (F, K, L) R3
G D
Scan
71
E, H
A, B, C
Scan
D, E, G, H
Scan
A
Scan
Column-Oriented Database Systems
R1 (A, B, C) R2 (D, E, G, H)
VLDB 2009 Tutorial
R1 (A, B, C) R2 (D, E, G, H)
71
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Early Materialization Example
2 7 3 13 3 42 3 80 Construct (4,1,4) 12 1 11 1 6 1 2 1 2 3 3 3 7 13 42 80 1 Green 2 White 3 Brown Construct
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
1 2 3
custID
Green White Brown
lastName
prodID storeID quantity custID price
Facts
VLDB 2009 Tutorial
Customers
Column-Oriented Database Systems 72
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Early Materialization Example
7 White
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
13 Brown 42 Brown 80 Brown Join 2 7 3 13 3 42 3 80 1 Green 2 White 3 Brown
VLDB 2009 Tutorial
Column-Oriented Database Systems
73
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Late Materialization Example
1 2 3 4 Join 2 3 1 3
QUERY: SELECT C.lastName,SUM(F.price) FROM facts AS F, customers AS C WHERE F.custID = C.custID GROUP BY C.lastName
(4,1,4)
12 1 11 1
6 1 2 1
2 3 1 3
7 13 42 80
1 2 3
custID
Green White Brown
lastName
Late materialized join causes out of order probing of projected columns from the inner relation
prodID
storeID quantity custID price
Facts
VLDB 2009 Tutorial
Customers
Column-Oriented Database Systems 74
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Late Materialized Join Performance
l
Naïve LM join about 2X slower than EM join on typical queries (due to random I/O)
l
This number is very dependent on
l l l
Amount of memory available Number of projected attributes Join cardinality
l
But we can do better
l l l
Invisible Join Jive/Flash Join Radix cluster/decluster join
VLDB 2009 Tutorial
Column-Oriented Database Systems
75
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
l
Designed for typical joins when data is modeled using a star schema
l
One (“fact”) table is joined with multiple dimension tables select c_nation, s_nation, d_year, sum(lo_revenue) as revenue from customer, lineorder, supplier, date where lo_custkey = c_custkey and lo_suppkey = s_suppkey and lo_orderdate = d_datekey and c_region = 'ASIA‘ and s_region = 'ASIA‘ and d_year >= 1992 and d_year <= 1997 group by c_nation, s_nation, d_year order by d_year asc, revenue desc;
l
Typical query:
VLDB 2009 Tutorial
Column-Oriented Database Systems
76
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
custkey region nation 1 ASIA CHINA 2 ASIA INDIA 3 ASIA INDIA 4 EUROPE FRANCE
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
Apply “region = ‘Asia’” On Customer Table
… … … … …
Hash Table (or bit-map) Containing Keys 1, 2 and 3
Apply “region = ‘Asia’” On Supplier Table
suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 ASIA JAPAN … … … …
Hash Table (or bit-map) Containing Keys 1, 3
Apply “year in [1992,1997]” On Date Table
dateid 01011997 01021997 01031997 year 1997 1997 1997 … … … …
Hash Table Containing Keys 01011997, 01021997, and 01031997
Column-Oriented Database Systems 77
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Original Fact Table
orderkey custkey suppkey orderdate revenue 1 3 1 01011997 43256 2 3 2 01011997 33333 3 4 3 01021997 12121 4 1 1 01021997 23233 5 4 2 01021997 45456 6 1 2 01031997 43251 7 3 2 01031997 34235
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi et. al. SIGMOD 2008. 1 1 1 1 0 1 0 1 1 1 1 1 0 0 1 1 0 1 1 0 1
& & = +
1 0 0 1 0 0 0
Hash Table Containing Keys 1, 2 and 3
custkey 3 3 4 1 4 1 3
VLDB 2009 Tutorial
+
Hash Table Containing Keys 1 and 3
1 1 0 1 0 1 1
suppkey 1 2 3 1 2 2 2
+
Hash Table Containing Keys 01011997, 01021997, and 01031997
1 0 1 1 0 0 0
orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
1 1 1 1 1 1 1
78
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
custkey 3 3 4 1 4 1 3 suppkey 1 2 3 1 2 2 2 orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
+ + +
1 0 0 1 0 0 0
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
1997 1997 1997
INDIA CHINA
1 0 0 1 0 0 0
1 1
RUSSIA SPAIN JAPAN
RUSSIA RUSSIA
1 0 0 1 0 0 0
01011997 01021997
JOIN
01011997 01021997 01031997
=
1997 1997
VLDB 2009 Tutorial
Column-Oriented Database Systems
79
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
custkey region nation 1 ASIA CHINA 2 ASIA INDIA 3 ASIA INDIA 4 EUROPE FRANCE
“Column-Stores vs Row-Stores: How Different are They Really?” Abadi, Madden, and Hachem. SIGMOD 2008.
Apply “region = ‘Asia’” On Customer Table
… … … … …
Hash Table (or bit-map) Containing Keys 1, 2 and 3
Range [1-3]
(between-predicate rewriting)
Apply “region = ‘Asia’” On Supplier Table
suppkey region nation 1 ASIA RUSSIA 2 EUROPE SPAIN 3 ASIA JAPAN … … … …
Hash Table (or bit-map) Containing Keys 1, 3
Apply “year in [1992,1997]” On Date Table
dateid 01011997 01021997 01031997 year 1997 1997 1997 … … … …
Hash Table Containing Keys 01011997, 01021997, and 01031997
Column-Oriented Database Systems 80
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Invisible Join
l
Bottom Line
l
l
l
l
l
l
Many data warehouses model data using star/snowflake schemes Joins of one (fact) table with many dimension tables is common Invisible join takes advantage of this by making sure that the table that can be accessed in position order is the fact table for each join Position lists from the fact table are then intersected (in position order) This reduces the amount of data that must be accessed out of order from the dimension tables “Between-predicate rewriting” trick not relevant for this discussion
Column-Oriented Database Systems 81
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
custkey 3 3 4 1 4 1 3 suppkey 1 2 3 1 2 2 2 orderdate 01011997 01011997 01021997 01021997 01021997 01031997 01031997
+ + +
1 0 0 1 0 0 0
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
1997 1997 1997
INDIA CHINA
Still accessing table out of order
1 0 0 1 0 0 0 1 1
RUSSIA SPAIN JAPAN
RUSSIA RUSSIA
1 0 0 1 0 0 0
01011997 01021997
JOIN
01011997 01021997 01031997
=
1997 1997
VLDB 2009 Tutorial
Column-Oriented Database Systems
82
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
3 1
+
CHINA INDIA INDIA FRANCE
=
INDIA CHINA
“Fast Joins using Join Indices”. Li and Ross, VLDBJ 8:1-24, 1999.
Still accessing table out of order
“Query Processing Techniques for Solid State Drives”. Tsirogiannis, Harizopoulos et. al. SIGMOD 2009.
VLDB 2009 Tutorial
Column-Oriented Database Systems
83
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
1.
2.
3.
4.
Add column with dense ascending integers from 1 Sort new position list by second column Probe projected column in order using new sorted position list, keeping first column from position list around Sort new result by first column
1 2
3 1
+ +
CHINA INDIA INDIA FRANCE
= =
2 1
INDIA CHINA
2 1
1 3
CHINA INDIA INDIA FRANCE
CHINA INDIA
1 2
INDIA CHINA
VLDB 2009 Tutorial
Column-Oriented Database Systems
84
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Jive/Flash Join
l
Bottom Line
l
Instead of probing projected columns from inner table out of order:
l l l
Sort join index Probe projected columns in order Sort result using an added column LM has the extra sorts (EM accesses all columns in order) LM only has to fit join columns into memory (EM needs join columns and all projected columns)
§
l
LM vs EM tradeoffs:
l l
Results in big memory and CPU savings (see part 3 for why there is CPU savings)
l l
LM only has to materialize relevant columns In many cases LM advantages outweigh disadvantages
l
LM would be a clear winner if not for those pesky sorts … can we do better?
VLDB 2009 Tutorial
Column-Oriented Database Systems
85
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Radix Cluster/Decluster
l
The full sort from the Jive join is actually overkill
l
l l
We just want to access the storage blocks in order (we don’t mind random access within a block) So do a radix sort and stop early By stopping early, data within each block is accessed out of order, but in the order specified in the original join index
l
Use this pseudo-order to accelerate the post-probe sort as well
•“Database Architecture Optimized for the New Bottleneck: Memory Access” VLDB’99 •“Generic Database Cost Models for Hierarchical Memory Systems”, VLDB’02 (all Manegold, Boncz, Kersten)
“Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
VLDB 2009 Tutorial
Column-Oriented Database Systems
86
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Radix Cluster/Decluster
l
Bottom line
l
l
Both sorts from the Jive join can be significantly reduced in overhead Only been tested when there is sufficient memory for the entire join index to be stored three times
l
Technique is likely applicable to larger join indexes, but utility will go down a little Don’t want to use radix cluster/decluster if you have variablewidth column values or compression schemes that can only be decompressed starting from the beginning of the block
l
Only works if random access within a storage block
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
87
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
LM vs EM joins
l
l
Invisible, Jive, Flash, Cluster, Decluster techniques contain a bag of tricks to improve LM joins Research papers show that LM joins become 2X faster than EM joins (instead of 2X slower) for a wide array of query types
VLDB 2009 Tutorial
Column-Oriented Database Systems
88
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Tuple Construction Heuristics
l
l
For queries with selective predicates, aggregations, or compressed data, use late materialization For joins:
l
Research papers:
l
Always use late materialization Inner table to a join often materialized before join (reduces system complexity): Some systems will use LM only if columns from inner table can fit entirely in memory
l
Commercial systems:
l
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
89
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Outline
l
l l
Computational Efficiency of DB on modern hardware
how column-stores can help here Keynote revisited: MonetDB & VectorWise in more depth
l
l
CPU efficient column compression
vectorized decompression
l
l
Conclusions
future work
VLDB 2009 Tutorial
Column-Oriented Database Systems
90
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
40 years of hardware evolution vs. DBMS computational efficiency
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Architecture
Elements: l Storage
l
CPU caches L1/L2/L3
l l
Registers Execution Unit(s)
l l
Pipelined SIMD
VLDB 2009 Tutorial
Column-Oriented Database Systems
92
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
93
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
CPU Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
94
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DRAM Metrics
VLDB 2009 Tutorial
Column-Oriented Database Systems
95
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Super-Scalar Execution (pipelining)
VLDB 2009 Tutorial
Column-Oriented Database Systems
96
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Hazards
l
Data hazards
l l
l
Control Hazards
l l l
Dependencies between instructions L1 data cache misses
Branch mispredictions Computed branches (late binding) L1 instruction cache misses
Result: bubbles in the pipeline
Out-of-order execution addresses data hazards l control hazards typically more expensive
VLDB 2009 Tutorial Column-Oriented Database Systems 97
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
SIMD
l
Single Instruction Multiple Data
l l l
Same operation applied on a vector of values MMX: 64 bits, SSE: 128bits, AVX: 256bits SSE, e.g. multiply 8 short integers
Column-Oriented Database Systems 98
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
102 ivan
37 7
350
PROJECT next()
101 102 alice ivan 37 22 > 30 ? 22 TRUE 37 FALSE
SELECT id, name (age-30)*50 AS bonus FROM employee WHERE age > 30
SELECT next()
102 101 ivan alice 37 22
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
99
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
Operators
Iterator interface -open() -next(): tuple -close()
PROJECT next()
37 22 > 30 ?
SELECT next() SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
100
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
A Look at the Query Pipeline
102
ivan
350 37 *–50 7 30
next()
102 ivan
37 7
350
Primitives
Provide computational functionality All arithmetic allowed in expressions, e.g. Multiplication
7 * 50
PROJECT next()
101 102 alice ivan 37 22 > 30 ? 22 TRUE 37 FALSE
SELECT next()
102 101 ivan alice 37 22
SCAN
mult(int,int) Ł int
101
VLDB 2009 Tutorial
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Database Architecture causes Hazards
l
Data hazards
l l
l
Control Hazards
l l l
Dependencies between instructions L1 data cache misses Out-of-order Execution
SIMD
Branch mispredictions Computed branches (late binding) L1 instruction cache misses
work on one tuple at a time Large Tree/Hash Structures Code footprint of all operators in query plan exceeds L1 cache Data-dependent conditions PROJECT
Operators
Iterator interface -open() -next(): tuple -close()
SELECT
SCAN Next() late binding method calls “DBMSs On A Modern Processor: Where Does Time Go? ” Tree, List, Hash traversal Complex NSM record navigation Ailamaki, DeWitt, Hill, Wood, VLDB’99
VLDB 2009 Tutorial Column-Oriented Database Systems
102
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DBMS Computational Efficiency
TPC-H 1GB, query 1 l selects 98% of fact table, computes net prices and aggregates all l Results:
l l l
C program: MySQL: DBMS “X”:
? 26.2s 28.1s
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
VLDB 2009 Tutorial Column-Oriented Database Systems 103
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
DBMS Computational Efficiency
TPC-H 1GB, query 1 l selects 98% of fact table, computes net prices and aggregates all l Results:
l l l
C program: MySQL: DBMS “X”:
0.2s 26.2s 28.1s
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
VLDB 2009 Tutorial Column-Oriented Database Systems 104
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
MonetDB
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a column-store
l
l
“save disk I/O when scan-intensive queries columns” “avoid an expression interpreter to improve computational efficiency”
need a few
VLDB 2009 Tutorial
Column-Oriented Database Systems
106
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
SELECT FROM WHERE
id, name, (age-30)*50 as bonus people age > 30
VLDB 2009 Tutorial
Column-Oriented Database Systems
107
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
CPU happy? Give it “nice” code ! - few dependencies (control,data) - CPU gets out-of-order execution - compiler can e.g. generate SIMD One loop for an entire column batcalc_minus_int(int* res, - no per-tuple interpretation int* col, - arrays: no record navigation int val, - better instruction cache locality
int n) { for(i=0; i<n; i++) as bonus SELECT id, name, (age-30)*50 res[i] = col[i] - val; FROM people } WHERE age > 30
Simple, hardcoded semantics in operators
VLDB 2009 Tutorial
Column-Oriented Database Systems
108
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
RISC Database Algebra
CPU happy? Give it “nice” code ! - few dependencies (control,data) - CPU gets out-of-order execution - compiler can e.g. generate SIMD One loop for an entire column batcalc_minus_int(int* res, - no per-tuple interpretation int* col, - arrays: no record navigation int val, - better instruction cache locality
int n) { for(i=0; i<n; i++) as bonus SELECT id, name, (age-30)*50 res[i] = col[i] - val; FROM people } WHERE age > 30
MATERIALIZED intermediate results
VLDB 2009 Tutorial
Column-Oriented Database Systems
109
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a column-store
l l
“save disk I/O when scan-intensive queries
need a few columns”
“avoid an expression interpreter to improve computational efficiency” How?
l
RISC query algebra: hard-coded semantics
l
Decompose complex expressions in multiple operations
l
Operators only handle simple arrays
l
No code that handles slotted buffered record layout
l
Relational algebra becomes array manipulation language
l
Often SIMD for free Plus: use of cache-conscious algorithms for Sort/Aggr/Join
Column-Oriented Database Systems 110
l
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
a Faustian pact
l
You want efficiency
l
Simple hard-coded operators Result materialization
l
I take scalability
l
n
C program: MonetDB: MySQL: DBMS “X”:
0.2s 3.7s 26.2s 28.1s
n
n
n
VLDB 2009 Tutorial
Column-Oriented Database Systems
111
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
as a research platform
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
SIGMOD 1985
B MonetD bra lge BAT A ts suppor B MonetD G, L, ODM M SQL, X ..RDF
•“MIL Primitives for Querying a Fragmented World”, Boncz, Kersten, VLDBJ’98 •“Flattening an Object Algebra to Provide Performance” Boncz, Wilschut, Kersten, ICDE’98 •“MonetDB/XQuery: a fast XQuery processor powered C- by a relational engine” Boncz, Grust, vanKeulen, port on e p RDF su SW-Stor Rittinger, Teubner, SIGMOD’06 / •“SW-Store: a vertically partitioned DBMS for Semantic STORE Web data management“ Abadi, Marcus, Madden, Hollenbach, VLDBJ’09
Column-Oriented Database Systems 113
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
The
Extensible query lang frontend framework XQuery .. SGL? Extensible Dynamic
Software Stack
SQL 03 Optimizers MonetDB 5
RDF
Arrays
Runtime QOPT Framework!
SOAP MonetDB 4
MonetDB kernel
Extensible Architecture-Conscious compile Execution platform
OGIS
VLDB 2009 Tutorial
Column-Oriented Database Systems
114
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
optimizer
frontend
backend
as a research platform
l
Cache-Conscious Joins
l
•“Database Architecture Optimized for the New
l
MonetDB/XQuery:
l
Bottleneck: Memory Access” VLDB’99 Cost Models, Radix-cluster, •“Generic Database Cost Models for Hierarchical Memory Radix-decluster Systems”, VLDB’02 (all Manegold, Boncz, Kersten) •“Cache-Conscious Radix-Decluster Projections”, Manegold, Boncz, Nes, VLDB’04
structural joins exploiting positional column access
“MonetDB/XQuery: a fast XQuery processor powered by a relational engine” Boncz, Grust, vanKeulen, Rittinger, Teubner, SIGMOD’06 “Database Cracking”, CIDR’07 “Updating a cracked database “, SIGMOD’07 “Self-organizing tuple reconstruction in columnstores“, SIGMOD’09 (all Idreos, Manegold, Kersten)
l
Cracking:
l
on-the-fly automatic indexing without workload knowledge
l
Recycling:
l
“An architecture for recycling intermediates in a using materialized intermediates column-store”, Ivanova, Kersten, Nes, Goncalves, SIGMOD’09
l
Run-time Query Optimization:
l
correlation-aware run-time optimization without cost model
VLDB 2009 Tutorial
“ROX: run-time optimization of XQueries”, Abdelkader, Boncz, Manegold, vanKeulen, SIGMOD’09
115
Column-Oriented Database Systems
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
“MonetDB/X100” vectorized query processing
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB spin-off: MonetDB/X100 Materialization vs Pipelining
VLDB 2009 Tutorial
Column-Oriented Database Systems
117
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
next()
PROJECT next() SELECT next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
118
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
next()
> 30 ?
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
119
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
“Vectorized Observations: Processing”
In Cache
next()
102 ivan 350 104 peggy 750 - 30 * 50 102 ivan 37 7 350 104 peggy 45 15 750 > 30 ?
next() called much less often Ł more time spent in primitives vector = array of ~100 less in overhead processed in a tight loop primitive calls process an array of values cache Resident CPU in a loop:
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
120
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Observations: next() called much less often Ł more time spent in primitives less in overhead primitive calls process an array of values in a loop:
CPU Efficiency depends on “nice” code - out-of-order execution - few dependencies (control,data) - compiler support
102 ivan 350 104 peggy 750 - 30 * 50 102 ivan 37 7 350 104 peggy 45 15 750 > 30 ?
next()
PROJECT
next()
101 102 104 105 101 102 104 105
alice ivan peggy victor alice ivan peggy victor
22 37 45 25 22 37 45 25
FALSE TRUE TRUE FALSE SELECT
next()
Compilers like simple loops over arrays - loop-pipelining - automatic SIMD
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
121
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Observations: next() called much less often Ł more time spent in primitives less in overhead primitive calls process an array of values in a loop:
CPU Efficiency depends on “nice” code - out-of-order execution - few dependencies (control,data) - compiler support Compilers like simple loops over arrays - loop-pipelining - automatic SIMD
> 30 ? FALSE TRUE TRUE FALSE - 30 7 15
for(i=0; i<n; i++) res[i] = (col[i] > x) * 50
350 750
for(i=0; i<n; 30 ? > i++)
PROJECT
FALSE res[i] = (col[i] - x) TRUE TRUE FALSE SELECT for(i=0; i<n; i++)
* 50 350 750
res[i] = (col[i] * x)
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
122
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Tricks being played: - Late materialization - Materialization avoidance using selection vectors
> 30 ?
PROJECT
next()
1 2
SELECT next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
123
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
map_mul_flt_val_flt_col( float *res, int* sel, Tricks being played: float val, float *col, int n) - Late materialization next() { - Materialization avoidance for(int i=0; i<n; i++) using selection vectors res[i] = val * col[sel[i]]; } selection vectors used to reduce vector copying contain selected positions next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
- 30 * 50 7 15 350 750
> 30 ?
PROJECT
next()
1 2
SELECT
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
124
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
map_mul_flt_val_flt_col( float *res, int* sel, Tricks being played: float val, float *col, int n) - Late materialization next() { - Materialization avoidance for(int i=0; i<n; i++) using selection vectors res[i] = val * col[sel[i]]; } selection vectors used to reduce vector copying contain selected positions next()
101 102 104 105 alice ivan peggy victor 22 37 45 25
- 30 * 50 7 15 350 750
> 30 ?
PROJECT
next()
1 2
SELECT
SCAN
VLDB 2009 Tutorial
Column-Oriented Database Systems
125
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
MonetDB/X100
l
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Both efficiency
l
Vectorized primitives Pipelined query evaluation
l
and scalability..
l
n
C program: MonetDB/X100: MonetDB: MySQL: DBMS “X”:
0.2s 0.6s 3.7s 26.2s 28.1s
Column-Oriented Database Systems 126
n
n
n
n
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory Hierarchy
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 127
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Memory Hierarchy
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
Vectors are only the in-cache representation RAM & disk representation might actually be different
CPU cache
(vectorwise uses both PAX & DSM)
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 128
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Optimal Vector size?
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
X100 query engine
All vectors together should fit the CPU cache Optimizer should tune this, given the query characteristics.
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 129
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
Less and less iterator.next() and primitive function calls (“interpretation overhead”)
VLDB 2009 Tutorial
Column-Oriented Database Systems
130
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
Vectors start to exceed the CPU cache, causing additional memory traffic
VLDB 2009 Tutorial
Column-Oriented Database Systems
131
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Varying the Vector size
The benefit of selection vectors
VLDB 2009 Tutorial
Column-Oriented Database Systems
132
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
MonetDB/MIL materializes columns
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 133
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“MonetDB/X100: Hyper-Pipelining Query Execution ” Boncz, Zukowski, Nes, CIDR’05
Benefits of Vectorized Processing
l
100x less Function Calls
l
iterator.next(), primitives
“Buffering Database Operations for Enhanced Instruction Cache Performance” Zhou, Ross, SIGMOD’04
l
No Instruction Cache Misses
l
l
“Block oriented processing of relational database operations in Less Data Cache Misses modern computer architectures” l Cache-conscious data placement Padmanabhan, Malkemus, Agarwal, ICDE’01 No Tuple Navigation
l
High locality in the primitives
l
Primitives are record-oblivious, only see arrays
l
Vectorization allows algorithmic optimization
l
Move activities out of the loop (“strength reduction”)
l
Compiler-friendly function bodies
l
Loop-pipelining, automatic SIMD
VLDB 2009 Tutorial
Column-Oriented Database Systems
134
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Balancing Vectorized Query Execution with Bandwidth Optimized Storage ” Zukowski, CWI 2008
Vectorizing Relational Operators
l l
Project Select
l
Exploit selectivities, test buffer overflow
l
Aggregation
l
Ordered, Hashed
l
Sort
l
Radix-sort nicely vectorizes
l
Join
l
Merge-join + Hash-join
VLDB 2009 Tutorial
Column-Oriented Database Systems
135
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Efficient Column Store Compression
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Key Ingredients
l
l
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Compress relations on a per-column basis
Columns compress well
l
Decompress small vectors of tuples from a column into the CPU cache
l
Minimize main-memory overhead Exploit processing power of modern CPUs
l
Use light-weight, CPU-efficient algorithms
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
137
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Key Ingredients
l
l
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Compress relations on a per-column basis
Columns compress well
l
Decompress small vectors of tuples from a column into the CPU cache
l
Minimize main-memory overhead
VLDB 2009 Tutorial
Column-Oriented Database Systems
138
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Vectorized Decompression
X100 query engine
Idea: decompress a vector only compression: -between CPU and RAM -Instead of disk and RAM (classic)
CPU cache
RAM
ColumnBM (buffer manager)
(raid) Disk(s)
VLDB 2009 Tutorial Column-Oriented Database Systems 139
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
RAM-Cache Decompression
l
l
l
l
Decompress vectors on-demand into the cache RAM-Cache boundary only crossed once More (compressed) data cached in RAM Less bandwidth use
VLDB 2009 Tutorial
Column-Oriented Database Systems
140
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Multi-Core Bandwidth & Compression
Performance Degradation with Concurrent Queries
1.2
avg query speed (clock normalized)
1
0.8
0.6
0.4
Intel® Xeon E5410 (Harpertown) - Q6b Normal
0.2
Intel® Xeon E5410 (Harpertown) - Q6b Compressed Intel® Xeon X5560 (Nehalem) - Q6b Normal Intel® Xeon X5560 (Nehalem) - Q6b Compressed
0 1 2 3 4 5 6 7 8
number of concurrent queries
VLDB 2009 Tutorial Column-Oriented Database Systems 141
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
CPU Efficient Decompression
l l
Decoding loop over cache-resident vectors of code words Avoid control dependencies within decoding loop
l
no if-then-else constructs in loop body
l
Avoid data dependencies between loop iterations
VLDB 2009 Tutorial
Column-Oriented Database Systems
142
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Disk Block Layout
l
Forward growing section of arbitrary size code words (code word size fixed per block)
VLDB 2009 Tutorial
Column-Oriented Database Systems
143
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Disk Block Layout
l
l
Forward growing section of arbitrary size code words (code word size fixed per block) Backwards growing exception list
VLDB 2009 Tutorial
Column-Oriented Database Systems
144
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Naïve Decompression Algorithm
l
Use reserved value from code word domain (MAXCODE) to mark exception positions
VLDB 2009 Tutorial
Column-Oriented Database Systems
145
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Deterioriation With Exception%
l
1.2GB/s deteriorates to 0.4GB/s
Column-Oriented Database Systems 146
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Deterioriation With Exception%
l
Perf Counters: CPU mispredicts if-then-else
Column-Oriented Database Systems 147
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patching
l
Maintain a patch-list through code word section that links exception positions
VLDB 2009 Tutorial
Column-Oriented Database Systems
148
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patching
l
Maintain a patch-list through code word section that links exception positions After decoding, patch up the exception positions with the correct values
l
VLDB 2009 Tutorial
Column-Oriented Database Systems
149
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patched Decompression
VLDB 2009 Tutorial
Column-Oriented Database Systems
150
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Patched Decompression
VLDB 2009 Tutorial
Column-Oriented Database Systems
151
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
“Super-Scalar RAM-CPU Cache Compression” Zukowski, Heman, Nes, Boncz, ICDE’06
Decompression Bandwidth
Patching can be applied to: • Frame of Reference (PFOR) • Delta Compression (PFOR-DELTA) • Dictionary Compression (PDICT)
Makes these methods much more applicable to noisy data Ł without additional cost
l
Patching makes two passes, but is faster!
Column-Oriented Database Systems 152
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Column-Oriented Database Systems
Conclusion
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary (1/2)
l
Columns and Row-Stores: different?
l l
No fundamental differences Can current row-stores simulate column-stores now?
l
not efficiently: row-stores need change actually independent issues, on-the-fly conversion pays off column favors sequential access, row random Fractured mirrors PAX, Clotho Data morphing
l
On disk layout vs execution layout
l l
l
Mixed Layout schemes
l l l
VLDB 2009 Tutorial
Column-Oriented Database Systems
154
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Summary (2/2)
l
Crucial Columnar Techniques
l
Storage
l
Lean headers, sparse indices, fast positional access Operating on compressed data Lightweight, vectorized decompression Non-join: LM always wins Naïve/Invisible/Jive/Flash/Radix Join (LM often wins) Vectorized in-cache execution Exploiting SIMD
Column-Oriented Database Systems 155
l
Compression
l l
l
Late vs Early materialization
l l
l
Execution
l l
VLDB 2009 Tutorial
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Future Work
l
looking at write/load tradeoffs in column-stores
l
read-only vs batch loads vs trickle updates vs OLTP
VLDB 2009 Tutorial
Column-Oriented Database Systems
156
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (1/3)
l
Column-stores are update-in-place averse
l l l l
In-place: I/O for each column + re-compression + multiple sorted replicas + sparse tree indices
Update-in-place is infeasible!
VLDB 2009 Tutorial
Column-Oriented Database Systems
157
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (2/3)
l
Column-stores use differential mechanisms instead
l l l
Differential lists/files or more advanced (e.g. PDTs) Updates buffered in RAM, merged on each query Checkpointing merges differences in bulk sequentially
l
I/O trends favor this anyway
§ §
trade RAM for converting random into sequential I/O this trade is also needed in Flash (do not write randomly!) Depends on available RAM for buffering (how long until full) § Checkpoint must be done within that time § The longer it can run, the less it molests queries Using Flash for buffering differences buys a lot of time § Hundreds of GBs of differences per server
l
How high loads can it sustain?
§
§
VLDB 2009 Tutorial
Column-Oriented Database Systems
158
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Updates (3/3)
l l
Differential transactions favored by hardware trends Snapshot semantics accepted by the user community
l
can always convert to serialized
“Serializable Isolation For Snapshot Databases” Alomari, Cahill, Fekete, Roehm, SIGMOD’08
Ł
Row stores could also use differential transactions and be efficient!
Ł Ł
Implies a departure from ARIES Implies a full rewrite
My conclusion: a system that combines row- and columns needs differentially implemented transactions. Starting from a pure column-store, this is a limited add-on. Starting from a pure row-store, this implies a full rewrite.
VLDB 2009 Tutorial Column-Oriented Database Systems 159
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Future Work
l
looking at write/load tradeoffs in column-stores
l
read-only vs batch loads vs trickle updates vs OLTP
l l
database design for column-stores column-store specific optimizers
l
compression/materialization/join tricks Ł cost models? can row-stores learn new column tricks?
l
l
hybrid column-row systems
l
l
Study of the minimal number changes one needs to make to a row store to get the majority of the benefits of a column-store Alternative: add features to column-stores that make them more like row stores
VLDB 2009 Tutorial
Column-Oriented Database Systems
160
Re-use permitted when acknowledging the original © Stavros Harizopoulos, Daniel Abadi, Peter Boncz (2009)
Conclusion
l
Columnar techniques provide clear benefits for:
l l
Data warehousing, BI Information retrieval, graphs, e-science Without these, existing row systems do not benefit Row-stores may adopt some column-store techniques Column-stores add row-store (or PAX) functionality
l
A number of crucial techniques make them effective
l
l
Row-Stores and column-stores could be combined
l l
l
Many open issues to do research on!
VLDB 2009 Tutorial
Column-Oriented Database Systems
161
