TPC-DS Benchmark Report

TPC-DS is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). TPC-DS models several generally applicable aspects of a decision support system, including queries and data maintenance. TPC-DS aims to provide a comprehensive and realistic workload for testing and evaluating the performance of database systems in a retail environment. The TPC-DS benchmark simulates the sales and return data of three sales channels (stores, Internet, and catalog) in a retail enterprise. In addition to creating tables for sales and return data models, it also includes a simple inventory system and a promotion system.

The benchmark tested a total of 99 complex SQL queries against 24 tables. The test utilized a 1TB data volume, and the primary performance metric was the response time of each query, that is, the duration between submitting the query and returning the results.

The following section presents the performance of three databases in single concurrency and 5-concurrency scenarios. The time unit is seconds, which measured the total time required for all SQL statements to complete.

Database	Total Execution Time in Single Concurrency	Total Execution Time in 5-Concurrency
SynxDB	5335	21125
Greenplum 6	6834	28255
Greenplum 7	6088	24750

Disclaimer

Greenplum® is a registered trademark of Broadcom Inc. Synx Data Labs and SynxDB are not affiliated with, endorsed by, or sponsored by Broadcom Inc. Any references to Greenplum are for comparative, educational, and interoperability purposes only.

Test environment

Hardware

In this test, we used AWS platform compute instances. To ensure fairness and comparability of the test environment, identical hardware resources were allocated to SynxDB and Greenplum, enabling accurate comparison between them.

Instance Type	Number of Instance	Region
r5.4xlarge	5	us-east-2a

The configuration of a single instance was as follows:

Processors	Memory	Storage	Network
16 cores	128 GB	8 * 500GB EBS (gp3)	10 Gbit/s

Total resources were as follows:

Total Processors	Total Memory	Total Storage	Total Network
80 cores	640 GB	20000 GB	50 Gbit/s

Software

Based on the aforementioned hardware resources, both SynxDB and Greenplum were deployed in a multi-node manner. 8 primary segments and 8 mirrors were deployed on each instance (mirror-mode=grouped), while the coordinator nodes and data nodes were deployed independently. The deployment architecture diagram is as follows:

Operating system and database kernel information:

Item	SynxDB	Greenplum
Operating system version	Rocky Linux 9.5	Rocky Linux 9.5
Database version	SynxDB 4.0	OSS Greenplum 6.27.1 OSS Greenplum 7.1.0

Test program

Tests were conducted on both SynxDB and Greenplum with a data scale of 1TB. All test results were obtained through actual measurements by SynxDB engineers using the same machine configuration and database parameters. The following methods were adopted during the testing process:

• Used TPC-DS benchmark test tool.

• Data storage method: AOCS columnar storage.

• Data compression method and level: Used zstd compression with compression level 1.

Test steps

1. Prepare the test environment.

   1. Install the SynxDB 4.0 test cluster following the SynxDB official documentation.

   2. Download and install Greenplum 6.27.1 and Greenplum 7.1.0.

2. Set parameters. Both SynxDB and Greenplum were configured with tunable parameters and options different from the defaults. The database tuning options were as follows:

   Parameter name	SynxDB	Greenplum 6
   gp_vmem_protect_limit	15000	15000
   max_statement_mem	‘14GB’	‘14GB’
   statement_mem	‘12GB’	‘12GB’
   gp_fts_probe_timeout	600	600
   gp_fts_probe_interval	600	600
   gp_segment_connect_timeout	1800	1800
   gp_interconnect_queue_depth	10	10
   gp_interconnect_snd_queue_depth	8	8
   gp_interconnect_min_retries_before_timeout	100	100
   gp_autostats_mode	‘none’	‘none’
   autovacuum	off	off
   max_connections	100/500	100/500
   max_prepared_transactions	100	100
   gp_enable_runtime_filter_pushdown	on	/

3. Download the TPC-DS benchmark test tool from https://github.com/HashDataFE/TPC-DS-CBDB.

4. Run the run.sh script.

5. Check the execution results through the tpcds_%y%m%d_%H%M%S.log log file.

Test conclusion

Multiple tests were conducted on both SynxDB and OSS Greenplum, and the best results were selected for performance comparison.

Data loading performance

The following table shows the comparison of loading TPC-DS data tables with a data volume of 1TB.

Database	Loading Speed	Time Required to Load 1TB of Data
SynxDB	700(MB/s)	25 (minutes)
Greenplum 6	700(MB/s)	25 (minutes)
Greenplum 7	700(MB/s)	25 (minutes)

Performance testing of 99 queries

The comparison of the 99 SQL statements of TPC-DS was as follows:

SynxDB (s)	Greenplum 6 (s)	Greenplum 7 (s)
5335	6834	6088

Conclusion: In terms of sequential execution, SynxDB was approximately 21.9% faster than Greenplum 6 and about 12.3% faster than Greenplum 7.

The details of SQL execution time were as follows:

NO.	SynxDB (s)	Greenplum 6 (s)	Greenplum 7 (s)
Q01	5	8	6
Q02	45	83	45
Q03	10	18	17
Q04	179	207	184
Q05	42	66	50
Q06	11	17	17
Q07	26	37	35
Q08	14	20	19
Q09	138	212	157
Q10	20	33	29
Q11	106	119	109
Q12	3	5	4
Q13	38	60	47
Q14	314	347	314
Q15	7	11	10
Q16	17	47	29
Q17	24	54	41
Q18	17	26	26
Q19	14	23	20
Q20	5	8	8
Q21	6	11	11
Q22	26	29	33
Q23	677	781	707
Q24	203	236	239
Q25	26	53	39
Q26	13	18	18
Q27	24	34	35
Q28	73	119	99
Q29	28	50	38
Q30	4	3	3
Q31	30	41	37
Q32	9	17	15
Q33	22	35	35
Q34	17	27	24
Q35	29	44	41
Q36	32	37	35
Q37	7	16	12
Q38	59	72	68
Q39	29	34	31
Q40	8	15	11
Q41	1	1	1
Q42	10	19	18
Q43	19	29	23
Q44	33	41	50
Q45	5	7	6
Q46	23	32	31
Q47	54	61	52
Q48	33	55	39
Q49	42	57	48
Q50	15	33	32
Q51	47	56	50
Q52	10	18	16
Q53	17	22	21
Q54	17	39	37
Q55	10	18	16
Q56	21	34	32
Q57	26	28	24
Q58	19	31	29
Q59	113	176	109
Q60	23	35	36
Q61	33	56	43
Q62	12	15	11
Q63	14	24	22
Q64	75	89	109
Q65	69	82	72
Q66	18	23	21
Q67	938	892	883
Q68	24	33	31
Q69	18	32	27
Q70	59	62	64
Q71	23	34	32
Q72	54	278	57
Q73	15	22	20
Q74	88	85	80
Q75	58	78	76
Q76	21	24	24
Q77	29	41	37
Q78	129	227	194
Q79	34	38	41
Q80	41	64	49
Q81	6	6	5
Q82	18	21	19
Q83	3	5	4
Q84	2	4	3
Q85	10	17	13
Q86	11	12	11
Q87	58	70	68
Q88	83	165	162
Q89	18	27	24
Q90	6	10	8
Q91	1	4	2
Q92	6	10	9
Q93	19	34	32
Q94	21	23	14
Q95	325	234	338
Q96	11	23	22
Q97	57	60	56
Q98	12	18	16
Q99	22	28	22

Throughput testing in TPC-DS scenario

The performance comparison of 99 SQL statements of TPC-DS in a 5-concurrency scenario is as follows:

SynxDB (s)	Greenplum 6 (s)	Greenplum 7 (s)
21125	28255	24750

Conclusion: In the 5-concurrency scenario, SynxDB was 25.2% faster than Greenplum 6 and 14.6% faster than Greenplum 7. In multi-concurrency scenarios, SynxDB outperformed Greenplum 6 and Greenplum 7 in terms of performance.

The detailed data was as follows:

NO.	SynxDB (s)					Greenplum 6 (s)					Greenplum 7 (s)
	S1	S2	S3	S4	S5	S1	S2	S3	S4	S5	S1	S2	S3	S4	S5
Q01	23	21	61	36	22	41	45	18	21	28	25	36	31	47	17
Q02	259	223	211	132	197	323	266	371	227	336	166	149	228	314	203
Q03	27	52	38	25	31	60	92	107	54	79	66	88	74	55	57
Q04	803	734	867	539	685	949	1018	912	548	985	734	1055	1153	626	670
Q05	88	91	107	99	98	152	231	202	177	159	106	130	154	138	118
Q06	81	44	45	52	74	74	88	79	78	103	62	82	83	113	97
Q07	142	89	88	113	105	163	129	114	171	99	160	101	108	154	86
Q08	51	58	61	43	67	64	69	85	63	108	75	91	81	50	104
Q09	225	197	233	178	234	269	246	243	240	224	195	186	192	174	161
Q10	56	52	54	58	41	94	189	92	100	60	122	81	105	67	80
Q11	518	473	903	845	809	395	527	1195	557	768	546	523	1113	700	745
Q12	7	16	12	13	14	18	35	15	14	27	20	43	16	31	17
Q13	116	158	143	185	107	150	362	269	394	268	180	237	151	244	237
Q14	818	974	707	691	771	672	954	704	647	802	790	981	681	635	804
Q15	18	24	19	20	21	34	35	33	32	25	28	32	32	28	39
Q16	119	61	58	50	53	576	558	385	170	373	106	115	110	81	295
Q17	92	87	89	99	153	136	123	368	220	100	128	142	243	130	121
Q18	65	53	66	102	63	116	70	66	87	51	66	87	70	137	47
Q19	78	51	57	63	82	99	91	75	148	97	92	93	69	88	115
Q20	14	17	18	18	20	25	28	24	28	23	38	30	40	39	38
Q21	21	10	38	8	22	29	24	24	20	37	39	31	22	25	56
Q22	52	106	83	46	89	266	65	79	70	52	202	62	57	62	54
Q23	3222	3287	3186	4500	3269	3470	3726	3697	5115	2665	3094	3226	2819	4791	2694
Q24	971	1024	1385	809	569	932	1171	1699	677	517	1078	1229	1432	831	579
Q25	111	71	89	93	84	248	169	172	164	163	205	107	132	114	122
Q26	40	47	94	43	47	82	163	86	62	44	61	49	60	58	63
Q27	115	176	118	95	85	291	200	156	92	91	168	138	171	80	91
Q28	106	124	120	120	143	166	223	164	195	210	125	200	217	182	167
Q29	82	70	106	67	141	170	145	143	119	195	94	162	110	120	182
Q30	10	12	9	8	11	11	14	13	8	10	12	14	8	8	11
Q31	98	78	152	113	134	82	86	126	147	107	123	74	87	77	154
Q32	56	52	29	31	41	81	97	64	52	91	75	85	51	58	95
Q33	75	72	65	75	64	227	95	92	90	123	106	90	97	130	110
Q34	55	70	40	55	55	116	126	88	94	76	103	76	113	97	95
Q35	112	108	121	94	113	151	131	118	45	102	158	212	162	77	134
Q36	198	146	116	74	127	170	205	150	86	174	307	148	114	86	185
Q37	27	15	32	20	12	73	52	30	34	29	101	40	35	32	56
Q38	188	411	184	271	175	256	184	246	138	221	152	463	388	170	182
Q39	108	156	86	146	161	103	141	69	152	147	133	71	226	139	125
Q40	37	37	39	22	40	64	62	54	41	74	35	53	45	39	52
Q41	1	1	3	1	1	1	2	1	2	2	0	1	0	1	1
Q42	26	36	27	36	43	77	103	92	72	53	69	67	110	68	94
Q43	90	59	74	49	63	108	98	131	73	108	154	73	107	144	95
Q44	68	50	65	58	66	72	113	76	118	76	84	77	80	64	82
Q45	23	18	19	16	26	31	21	17	18	31	25	25	28	30	28
Q46	88	95	23	68	97	161	112	70	105	117	137	131	93	82	300
Q47	157	141	57	125	106	138	133	333	116	121	152	133	181	160	169
Q48	101	101	34	84	146	239	253	107	323	221	163	223	113	171	220
Q49	102	124	94	108	80	114	132	135	57	125	128	108	149	57	185
Q50	36	103	56	36	43	113	153	121	90	171	130	189	129	117	153
Q51	114	179	162	139	225	168	208	211	97	116	158	228	145	173	149
Q52	81	38	40	53	51	74	71	61	106	38	88	112	77	84	54
Q53	65	57	39	94	62	86	92	122	105	66	79	106	118	114	26
Q54	69	67	51	80	67	129	115	161	94	129	289	125	143	89	242
Q55	36	57	31	30	55	61	76	68	51	52	85	73	87	46	58
Q56	50	50	68	72	47	77	117	111	87	74	103	73	106	145	92
Q57	101	76	69	94	81	97	146	81	70	79	82	66	92	87	70
Q58	64	56	46	56	56	108	107	129	63	82	93	93	145	64	104
Q59	531	280	452	235	514	799	454	634	505	831	465	272	436	253	454
Q60	81	58	75	84	54	95	116	87	96	75	99	85	86	130	95
Q61	138	71	66	77	93	175	121	117	935	1267	113	115	123	115	131
Q62	41	34	40	51	33	53	48	59	69	47	52	45	31	57	83
Q63	79	89	58	76	42	101	132	134	101	62	116	117	158	109	89
Q64	324	233	322	255	272	370	269	258	457	233	379	337	320	259	288
Q65	367	177	164	233	207	282	249	227	166	191	207	217	225	192	322
Q66	55	67	17	43	51	53	89	37	47	55	53	81	52	45	78
Q67	3947	3853	4151	4255	4461	5552	5279	5484	7135	6889	4569	4224	4419	5219	5510
Q68	79	92	122	86	60	116	190	124	96	135	100	165	197	141	134
Q69	59	53	57	60	49	100	63	64	74	64	83	64	72	98	71
Q70	412	179	203	188	199	269	232	177	175	146	424	216	166	226	183
Q71	74	68	88	106	91	133	109	129	89	185	104	119	135	109	151
Q72	209	225	233	372	205	805	767	747	1012	1009	379	544	407	386	430
Q73	57	41	51	47	87	72	92	119	104	111	126	100	84	74	88
Q74	511	506	533	384	361	401	306	305	610	396	567	560	388	806	559
Q75	301	348	202	226	269	288	298	267	226	192	400	242	360	263	373
Q76	37	57	80	46	55	66	85	100	51	73	72	71	68	64	48
Q77	74	93	67	59	111	127	111	91	88	147	97	110	86	109	118
Q78	502	402	390	347	516	811	731	607	430	902	925	624	561	331	771
Q79	118	135	141	256	157	230	117	119	66	177	116	112	120	102	239
Q80	138	109	233	145	108	217	134	170	223	164	157	97	161	171	138
Q81	14	14	23	9	15	20	21	19	12	22	20	12	23	14	30
Q82	59	53	30	25	29	54	81	81	61	64	78	73	49	46	51
Q83	5	11	6	10	7	12	17	10	12	16	10	16	9	16	12
Q84	5	6	8	4	7	8	11	7	8	9	9	14	9	7	12
Q85	46	57	33	34	32	98	92	55	79	61	87	70	64	57	136
Q86	44	80	26	26	24	38	34	40	25	34	59	41	26	28	45
Q87	314	433	430	219	226	341	306	569	204	320	373	287	358	224	239
Q88	112	116	126	130	155	203	213	337	168	226	224	182	274	148	240
Q89	47	72	18	111	95	96	98	42	138	103	83	73	126	133	92
Q90	18	10	12	55	15	20	25	23	21	14	18	22	26	18	21
Q91	4	3	11	5	4	22	14	14	17	17	6	5	5	5	4
Q92	16	21	64	56	23	33	53	40	27	33	28	46	26	26	24
Q93	56	57	49	62	1	107	80	207	1	166	82	69	105	1	149
Q94	288	142	29	21	24	275	136	231	154	184	69	50	53	84	72
Q95	728	1522	1301	1056	1174	732	1472	1174	1201	1078	868	1781	1086	1384	1328
Q96	107	45	41	23	30	215	85	70	81	199	137	114	163	75	82
Q97	163	182	156	114	274	182	206	189	136	287	153	141	170	176	325
Q98	68	55	49	31	42	205	85	90	70	48	160	76	89	55	93
Q99	187	99	116	71	114	109	109	122	115	140	58	108	101	143	110