Introduction
Welcome to the future of Node.js data storage. jsondb-high bridges the gap between the simplicity of JSON files and the performance of enterprise-grade databases.
Features
โก Blazing Fast
Core logic in Rust via N-API. Up to 2M ops/s reads.
๐ Multi-Process Safe
OS-level advisory file locking prevents corruption.
๐งต Multi-Core
Adaptive parallelism utilizing all CPU cores with Rayon.
๐ก๏ธ ACID Durability
Write-Ahead Logging (WAL) ensures your data is safe.
Installation
# Using bun (recommended)
bun add jsondb-high
# Using npm
npm install jsondb-high
Requirements
- Node.js: >= 16.0.0
- Rust Toolchain: Optional โ auto-installed or WASM fallback
- Platforms: Linux, macOS, Windows (x64 & arm64)
Quick Start
import JSONDatabase from 'jsondb-high';
const db = new JSONDatabase('db.json');
// Write
await db.set('user.1', { name: 'Alice', role: 'admin' });
// Read
const user = await db.get('user.1');
console.log(user); // { name: 'Alice', role: 'admin' }
Hybrid Architecture
jsondb-high offers multiple storage and safety modes. Choose based on your performance and durability needs.
Durability Modes
const db = new JSONDatabase('db.json', {
durability: 'batched', // 'none' | 'lazy' | 'batched' | 'sync'
walFlushMs: 10, // Sync every 10ms (Group Commit)
lockMode: 'exclusive'
});
| Mode | Throughput | Latency | Durability Window |
|---|---|---|---|
none | ~545k ops/s | 0.002ms | Manual save only |
lazy | ~400k ops/s | 0.003ms | 100ms |
batched | ~525k ops/s | 0.002ms | 10ms (Recommended) |
sync | ~2k ops/s | 0.5ms | Immediate |
Locking Modes
Prevent corruption from multiple processes using the same file:
const db = new JSONDatabase('db.json', {
lockMode: 'exclusive' // 'exclusive' | 'shared' | 'none'
});
O(1) Indexing
const db = new JSONDatabase('db.json', {
indices: [{ name: 'email', path: 'users', field: 'email' }]
});
// Instant Lookup
const user = await db.findByIndex('email', 'alice@corp.com');
Parallel Processing
The database automatically detects available CPU cores and uses parallel processing for large datasets (โฅ100 items). Falls back to efficient single-threaded operation for small workloads to avoid overhead.
const info = db.getSystemInfo();
console.log(info);
// {
// availableCores: 8,
// parallelEnabled: true,
// recommendedBatchSize: 1000
// }
DBOptions Interface
interface DBOptions {
indices?: IndexConfig[]; // Array of index configurations
wal?: boolean; // Enable Write-Ahead Logging
encryptionKey?: string; // AES-256-GCM encryption key (32 chars)
autoSaveInterval?: number; // Auto-save interval in ms
lockMode?: 'exclusive' | 'shared' | 'none'; // Process locking mode
lockTimeoutMs?: number; // Lock timeout in ms
durability?: 'none' | 'lazy' | 'batched' | 'sync'; // Durability mode
walBatchSize?: number; // WAL batch size
walFlushMs?: number; // WAL flush interval in ms
schemas?: Record<string, Schema>; // Path-based schemas
slowQueryThresholdMs?: number; // Slow query threshold in ms
stripeCount?: number; // v6: DashMap concurrency stripes
bufferPoolSizeMB?: number; // v6: Buffer pool size for mmap
bufferPageSizeKB?: number; // v6: Page size for buffer pool
}
Basic Operations
await db.set('config.theme', 'dark');
const val = await db.get('config.theme', 'light');
if (await db.has('users.1')) { ... }
await db.delete('users.1.settings');
Array Operations
await db.push('users.1.tags', 'premium', 'beta');
await db.pull('users.1.tags', 'beta');
Math Operations
const newCount = await db.add('users.1.loginCount', 1);
const newCredits = await db.subtract('users.1.credits', 50);
Query Builder
const results = await db.query('users')
.where('age').gt(18)
.where('role').eq('admin')
.limit(10)
.sort({ age: -1 })
.select(['id', 'name'])
.exec();
Aggregations
const count = db.query('users').count();
const total = db.query('orders').sum('amount');
const grouped = db.query('users').groupBy('department');
Find Operations
const user = await db.find('users', u => u.age > 18);
const adults = await db.findAll('users', u => u.age >= 18);
Pagination
const page = await db.paginate('users', 1, 20);
Batch & Parallel
await db.batch([
{ type: 'set', path: 'logs.1', value: 'log data' },
{ type: 'delete', path: 'temp.cache' }
]);
// Parallel (High Performance)
await db.batchSetParallel(largeArrayOfOperations);
Transactions
await db.transaction(async (data) => {
if (data.bank.balance >= 100) {
data.bank.balance -= 100;
data.users['1'].wallet += 100;
}
return data;
});
Snapshots
const backupPath = await db.createSnapshot('daily');
await db.restoreSnapshot(backupPath);
TTL (Time to Live)
await db.setWithTTL('session.abc', { id: 1 }, 60);
Pub/Sub
const unsubscribe = db.subscribe('users.*', (newVal) => {
console.log('Changed:', newVal);
});
Encryption
const db = new JSONDatabase('secure.json', {
encryptionKey: 'your-32-character-secret-key!!'
});
Middleware
db.before('set', 'users.*', (ctx) => {
ctx.value.updatedAt = Date.now();
return ctx;
});
Utility Methods
await db.keys('users');
await db.values('users');
await db.count('users');
await db.clear();
await db.stats();
await db.save(); // Force save
await db.sync(); // Durability sync
Events
db.on('change', ({ path, value, oldValue }) => { ... });
db.on('batch', ({ operations }) => { ... });
db.on('transaction:commit', () => { ... });
db.on('transaction:rollback', ({ error }) => { ... });
db.on('snapshot:created', ({ path, name }) => { ... });
db.on('snapshot:restored', ({ path }) => { ... });
db.on('ttl:expired', ({ path }) => { ... });
db.on('error', (error) => { ... });
Schema Validation (v5.1+)
Define schemas to enforce data structure and validation rules at specific paths.
const db = new JSONDatabase('db.json', {
schemas: {
'users': {
type: 'object',
properties: {
id: { type: 'number' },
email: {
type: 'string',
pattern: '^[\\w.-]+@[\\w.-]+\\.\\w+$'
},
age: {
type: 'number',
minimum: 0, maximum: 150
},
role: {
type: 'string',
enum: ['admin', 'user', 'guest']
},
tags: {
type: 'array',
items: { type: 'string' },
uniqueItems: true
}
},
required: ['id', 'email']
}
}
});
// โ Missing required field
await db.set('users.1', { id: 1 });
// โ
Valid data
await db.set('users.1', {
id: 1,
email: 'alice@example.com',
age: 25,
role: 'admin'
});
Schema Types & Constraints
| Type | Constraints |
|---|---|
string | minLength, maxLength, pattern (regex) |
number | minimum, maximum, exclusiveMinimum, exclusiveMaximum |
array | minItems, maxItems, uniqueItems, items (item schema) |
object | properties, required |
| All types | enum (allowed values) |
Where Clauses
.where('field').eq(value) // Equal
.where('field').ne(value) // Not equal
.where('field').gt(value) // Greater than
.where('field').gte(value) // Greater or equal
.where('field').lt(value) // Less than
.where('field').lte(value) // Less or equal
.where('field').between(1, 10) // Between range
.where('field').in([1, 2]) // In array
.where('field').notIn([1]) // Not in array
.where('field').contains('x') // String contains
.where('field').startsWith('x')// String starts with
.where('field').endsWith('x') // String ends with
.where('field').matches(/^x/) // Regex match
.where('field').exists() // Field exists
.where('field').isNull() // Is null
.where('field').isNotNull() // Is not null
Parallel Query
High-performance filtering using native Rust parallel iteration.
const activeAdults = await db.parallelQuery('users', [
{ field: 'age', op: 'gte', value: 18 },
{ field: 'status', op: 'eq', value: 'active' }
]);
Parallel Aggregation
const count = await db.parallelAggregate('orders', 'count');
const totalRevenue = await db.parallelAggregate('orders', 'sum', 'amount');
const avgOrder = await db.parallelAggregate('orders', 'avg', 'amount');
Parallel Joins (Lookups)
const usersWithOrders = await db.parallelLookup(
'users', // left collection
'orders', // right collection
'id', // left field (users.id)
'userId', // right field (orders.userId)
'orders' // output field name
);
๐งฌ MVCC Concurrency (v6)
Multi-Version Concurrency Control ensures readers never block writers. Concurrent writes to the same key return a ConflictError instead of deadlocking.
const db = new JSONDatabase('db.json', {
stripeCount: 256, // DashMap auto-scales with CPU cores
});
// These proceed in parallel โ different collections, no locking
await Promise.all([
db.set('users.1', { name: 'Alice' }),
db.set('orders.1', { total: 99 }),
db.set('logs.1', { event: 'login' }),
]);
๐ mmap O(1) Startup (v6)
Database files are memory-mapped via memmap2 โ near-instant startup regardless of file size.
const db = new JSONDatabase('large.json', {
bufferPoolSizeMB: 128, // Configurable buffer pool
bufferPageSizeKB: 32, // Page size tuning
});
// 100MB file? Opens in <1ms via mmap
๐ฌ .explain() API (v6)
Inspect query execution plans โ no more guessing what the Rust engine is doing:
const plan = await db.query('users')
.where('age').gt(18)
.sort({ age: -1 })
.limit(10)
.explain();
// {
// scanType: 'FILTER_SCAN',
// collectionSize: 50000,
// filtersApplied: [{ field: 'age', op: 'Gt' }],
// sortApplied: [{ field: 'age', direction: 'desc' }],
// limit: 10,
// matchedCount: 1234,
// executionTimeMs: 0.42
// }
โช Point-in-Time Recovery (v6)
Create snapshots and restore to any previous state:
const snap = await db.createSnapshot('before-migration');
// ... make changes ...
await db.set('config.version', '7.0.0');
// Roll back!
await db.restoreSnapshot(snap);
๐ Pluggable Storage Adapters (v6)
Decouple the database from Node.js fs for browser/edge/serverless environments:
import { MemoryAdapter, FileSystemAdapter } from 'jsondb-high/adapters';
// In-memory (testing, ephemeral)
const mem = new MemoryAdapter();
await mem.write('{"test": 42}');
const data = await mem.read();
// Also: LocalStorageAdapter, IndexedDBAdapter, HttpAdapter
๐ก WAL Replication (v6)
Stream WAL entries to replicas with CRC32 integrity verification. Supports async and sync acknowledgment modes.
// WAL entries are automatically persisted
const db = new JSONDatabase('db.json', {
durability: 'batched',
walFlushMs: 5
});
await db.set('key', 'value');
await db.sync();
const status = db.walStatus();
// { enabled: true, committed_lsn: 42 }
๐ชถ Lite Mode (v6)
Stripped-down WASM-only build (<2MB) for AWS Lambda, edge functions, and browsers:
# Build the lite distribution
bun run build:lite
# Output: ./lite/ with WASM binary + shim + adapters only
# No native .node files, no rayon parallelism
# Perfect for size-constrained environments
๐ฆ WASM Fallback (v6)
100% installation success โ the package automatically falls back to WASM when no native binary is available and Rust is not installed:
// Loading priority:
// 1. Prebuild .node binary (prebuilds/index.linux-x64-gnu.node)
// 2. Build from source via cargo (if Rust installed)
// 3. WASM fallback (always works, slightly slower)
// You'll see this warning in WASM mode:
// [jsondb-high] Running in WASM fallback mode.
// Performance will be reduced (no parallelism).
TypeScript Interfaces
interface DBOptions {
indices?: IndexConfig[];
wal?: boolean;
encryptionKey?: string;
autoSaveInterval?: number;
lockMode?: 'exclusive' | 'shared' | 'none';
lockTimeoutMs?: number;
durability?: 'none' | 'lazy' | 'batched' | 'sync';
walBatchSize?: number;
walFlushMs?: number;
schemas?: Record<string, Schema>;
slowQueryThresholdMs?: number;
}
interface IndexConfig {
name: string;
path: string;
field: string;
}
Schema Types
interface Schema {
type: 'string' | 'number' | 'boolean' | 'array' | 'object';
properties?: Record<string, Schema>;
required?: string[];
items?: Schema;
enum?: any[];
minimum?: number;
maximum?: number;
minLength?: number;
maxLength?: number;
pattern?: string;
minItems?: number;
maxItems?: number;
uniqueItems?: boolean;
}
Query Types
interface ParallelFilter {
field: string;
op: 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte' |
'contains' | 'startswith' | 'endswith' |
'in' | 'notin' | 'regex';
value: any;
}
interface PaginateResult<T> {
data: T[];
meta: {
total: number;
pages: number;
page: number;
limit: number;
hasNext: boolean;
hasPrev: boolean;
};
}