In-Memory Analysis of Maritime Data Sets – A Database System for Efficiently Processing Very Large Data Sets in Maritime Domain Awareness and Operational Research

PDF

Authors
  1. Schaub, D. E.
Corporate Authors
Defence R&D Canada - Atlantic, Dartmouth NS (CAN)
Abstract
The present work develops an in-memory database system that enables efficient statistical calculations on extremely large sets of maritime data. The conventional approach of interspersing calculations with queries to relational databases incurs significant latencies when data retrieval patterns induce inefficient hard-disk operations. Mitigation through re-ordering data access is often feasible but at the expense of significant additional development involving practices in advanced computer programming. As random access of computer memory is several orders of magnitude faster than secondary storage access, it is highly desirable—when possible—to store databases exclusively in memory. Through bit-shifting, dynamic dictionary compression, and domain-specific data reduction techniques, the present work demonstrates a functional data storage and processing system capable of storing the entire Global Positional Warehouse in the memory of a personal computer. Data are made available to the user through a simple application programming interface that allows rapid development of efficient, large-scale statistical analysis programs.

Il y a un résumé en français ici.

Keywords
Maritime Situational Awareness (MSA);Maritime Domain Awareness (MDA);Data Analysis;Database, Operations Centre;Operations Research (OR);Anomaly Detection;Data Mining;Pattern of Life
Report Number
DRDC-ATLANTIC-TM-2013-211 — Technical Memorandum
Date of publication
01 Apr 2014
Number of Pages
36
DSTKIM No
CA046340
CANDIS No
806728
Format(s):
Electronic Document(PDF)

Permanent link

Document 1 of 1

Date modified: