# Corelib Monorepo

[![CI/CD Pipeline](https://github.com/ckir/corelib/actions/workflows/pipeline.yml/badge.svg)](https://github.com/ckir/corelib/actions/workflows/pipeline.yml)
[![npm version](https://img.shields.io/npm/v/@ckirg/corelib.svg?label=@ckirg/corelib)](https://www.npmjs.com/package/@ckirg/corelib)
[![npm version](https://img.shields.io/npm/v/@ckirg/corelib-markets.svg?label=@ckirg/corelib-markets)](https://www.npmjs.com/package/@ckirg/corelib-markets)
[![npm version](https://img.shields.io/npm/v/@ckirg/corelib-cloud.svg?label=@ckirg/corelib-cloud)](https://www.npmjs.com/package/@ckirg/corelib-cloud)
[![License](https://img.shields.io/github/license/ckir/corelib.svg)](https://github.com/ckir/corelib/blob/main/LICENSE)
[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
[![Rust](https://img.shields.io/badge/Rust-1.82+-orange.svg)](https://www.rust-lang.org/)
[![pnpm](https://img.shields.io/badge/pnpm-11.5+-yellow.svg)](https://pnpm.io/)
[![Node.js](https://img.shields.io/badge/Node.js-24+-green.svg)](https://nodejs.org/)
[![Bun](https://img.shields.io/badge/Bun-1.1+-yellow.svg)](https://bun.sh/)
[![Deno](https://img.shields.io/badge/Deno-2.0+-black.svg)](https://deno.land/)

A high-performance, resilient, and multi-runtime monorepo for TypeScript and Rust. This workspace provides foundational utilities, cloud extensions, and financial market tooling designed for **Node.js, Bun, and Deno**.

## 🚀 Overview

This repository is structured as a pnpm monorepo, integrating TypeScript's flexibility with Rust's performance through FFI (Foreign Function Interface). It is built on principles of resilience, performance, and cross-runtime compatibility.

### Key Features
- **Isomorphic Core**: All foundational utilities in `ts-core` support Node.js, Bun, and Deno.
- **Resilient HTTP**: Standardized fetch wrapper (`RequestUnlimited`) with automatic retries and consistent error serialization. `RequestProxied` extends this with self-healing, rotating proxy pools that automatically prune dead endpoints.
- **Strict Logging**: Structured logging API with `(msg: string, extras?: object)` signature and telemetry support.
- **Native Rust Performance**: Performance-critical paths (streaming engines, WS clients) implemented in Rust via N-API/FFI — zero JS event-loop blocking via napi worker threads.
- **Real-Time Streaming**: Live market data from Alpaca, Finnhub, and Yahoo Finance; the Rust engine owns the WebSocket, delivering data to JS as events via napi `ThreadsafeFunction`.
- **Unified Configuration**: Centralized `ConfigManager` supporting overrides via files, environment variables, and CLI.

---

## Architecture

```mermaid
graph TD
    A[Your App] -->|npm install| B["@ckirg/corelib (ts-core)"]
    A -->|npm install| C["@ckirg/corelib-markets (ts-markets)"]
    A -->|npm install| D["@ckirg/corelib-cloud (ts-cloud)"]
    C --> B
    D --> B
    D --> C
    B <-->|N-API FFI| E["corelib-rust (.node binary)"]
    C -->|"streaming via FFI\n(napi ThreadsafeFunction)"| E
    E -->|"WS: Alpaca / Finnhub / Yahoo"| F[Live Market Feeds]
```

> Full detail: [ARCHITECTURE.md](./ARCHITECTURE.md) — package map, runtime matrix, FFI
> surface, proxy semantics, CI/CD stages.

---

## 📂 Project Structure & Packages

| Package | Description | Documentation |
| :--- | :--- | :--- |
| **[`@ckirg/corelib`](./ts-core/README.md)** | Core logic, FFI bridge, resilient HTTP, and database abstractions. | [README](./ts-core/README.md) |
| **[`@ckirg/corelib-markets`](./ts-markets/README.md)** | Market data tooling (Nasdaq, Yahoo) and financial indicators. | [README](./ts-markets/README.md) |
| **[`@ckirg/corelib-cloud`](./ts-cloud/README.md)** | Cloud-specific extensions for AWS, GCP, and Cloudflare. | [README](./ts-cloud/README.md) |
| **[`corelib-rust`](./rust/README.md)** | Native Rust core exposed via N-API (FFI). | [README](./rust/README.md) |

---

## 📦 Installation for External Projects

These packages are published to the **public npm registry** under the `@ckirg` scope.

### 1. Install
```bash
npm install @ckirg/corelib @ckirg/corelib-markets
# or: pnpm add @ckirg/corelib @ckirg/corelib-markets
# or: bun add  @ckirg/corelib @ckirg/corelib-markets
```

### 2. Required `.npmrc` (when using `@ckirg/corelib-markets`)
`@ckirg/corelib-markets` depends on `@gadicc/yahoo-finance2`, distributed via **JSR**. Add the JSR registry mapping to your project's `.npmrc`, or installs will 404 on the transitive `@jsr/...` package:
```
@jsr:registry=https://npm.jsr.io
```

### 🦀 The Native Rust Binary
`@ckirg/corelib` ships a `postinstall` that downloads the correct prebuilt Rust binary (`corelib-rust-*.node`) for your OS/arch from the matching GitHub Release.

- **Supported targets:** `linux-x64`, `linux-arm64`, `win32-x64`, `darwin-x64`, `darwin-arm64`. The Linux modules are built against glibc 2.34, so they also load on AWS Lambda and older distros (`linux-arm64`, for AWS Graviton, Ampere or a Raspberry Pi, ships from v0.3.1; v0.3.0's `linux-x64` needs glibc 2.38).
- **Bun consumers:** Bun blocks dependency `postinstall` scripts by default. Add corelib to `trustedDependencies` in your `package.json`, or the binary is never fetched and the FFI fails at runtime:
  ```json
  "trustedDependencies": ["@ckirg/corelib"]
  ```
- **`MODE=development` caveat:** the postinstall **skips** the download when `MODE=development` (or under CI's `GITHUB_ACTIONS`). Don't set `MODE=development` in a consuming app's `.env`, or you'll hit a "module not found" for the addon.
- **Manual trigger** if the download was skipped or blocked:
  ```bash
  node node_modules/@ckirg/corelib/scripts/postinstall.js
  ```

---

## 🛠️ Getting Started

### Prerequisites
- [Node.js](https://nodejs.org/) (v24+ recommended)
- [pnpm](https://pnpm.io/) (v11.5+ required)
- [Rust](https://rust-lang.org/) (for native module builds)

### Common Commands
All workspace-wide commands are managed through `pnpm`:

| Command | Description |
| :--- | :--- |
| `pnpm install` | Install all dependencies and link packages |
| `pnpm build-all` | Build all packages in correct order |
| `pnpm test-all` | Run all unit and integration tests |
| `pnpm lint-all` | Run Biome linting and formatting checks |
| `pnpm format-all` | Apply Biome formatting automatically |
| `pnpm docs-all` | Regenerate all API documentation |
| `pnpm clean-all` | Wipe all `node_modules`, `dist`, and `target` folders |

---

## 📖 Usage Examples

### 1. Core Utilities (`@ckirg/corelib`)
```typescript
import { logger, endPoint, RequestProxied, ConfigManager } from '@ckirg/corelib';

// 1. Resilient Fetch (ky-powered with retries)
const result = await endPoint('https://api.nasdaq.com/api/market-info');

// 2. Proxied Fetch (Automatic rotation & fallback)
const client = new RequestProxied(["https://proxy1.com", "https://proxy2.com"]);
const proxiedResult = await client.endPoint("https://target.com");

// 3. Configuration Management
const config = ConfigManager.getInstance();
await config.initialize();              // single-flight; safe under concurrency
if (!config.isInitialized) await config.whenReady();
const port = config.get("database.port"); // read at the use site, don't cache slices
```

### 2. Market Data (`@ckirg/corelib-markets`)
```typescript
import { MarketMonitor, MarketSymbols, Historical, type MarketPhase } from '@ckirg/corelib-markets';

// 1. Resilient Status Poller
const monitor = new MarketMonitor();
monitor.on("status-change", (phase: MarketPhase) => {
  console.log(`Market phase changed to ${phase}`);
});
monitor.start();

// 2. Persistent Symbol Database
const symbols = new MarketSymbols();
const aapl = await symbols.get("AAPL"); // Auto-refreshes if needed

// 3. Resilient Historical Data (Yahoo Finance v3)
const history = await Historical.getData("AAPL", { period1: "2023-01-01" });
if (history.status === 'success') {
  console.log(`Retrieved ${history.value.length} quotes`);
}
```

### 2b. Real-Time Streaming (flagship — Rust WS engine, zero event-loop blocking)
```typescript
import { AlpacaStreaming } from '@ckirg/corelib-markets';

const stream = new AlpacaStreaming();

// Credentials via init() or APCA_API_KEY_ID / APCA_API_SECRET_KEY env vars
await stream.init({ keyId: 'YOUR_KEY', secretKey: 'YOUR_SECRET' });
await stream.start();

stream.subscribe(['AAPL', 'TSLA', 'NVDA']);

stream.on('pricing', (data) => {
  console.log(`${data.symbol}: $${data.price}`);
});
stream.on('connected',    () => console.log('connected'));
stream.on('disconnected', () => console.log('disconnected'));
stream.on('error',        (e) => console.error('error', e));

// FinnhubStreaming and YahooStreaming follow the same EventEmitter pattern
// (Finnhub uses token auth; Yahoo is unauthenticated — see ts-markets/README.md)
```

### 3. Edge Proxy Services (`@ckirg/corelib-cloud`)
A portable TypeScript service exposing corelib logic on **Cloudflare Workers**, **AWS Lambda**, and **Cloud Run**.

```bash
# Example: Fetching market data via the Edge Proxy
curl -X POST https://your-edge-service/api/v1/markets/nasdaq \
     -H "Content-Type: application/json" \
     -d '{
       "url": "https://api.nasdaq.com/api/quote/AAPL/info?assetclass=stocks"
     }'
```

---

## 📖 Documentation

The project publishes a documentation site at [ckir.github.io/corelib](https://ckir.github.io/corelib/):

- **[Landing site](https://ckir.github.io/corelib/)**
- **[API Documentation Index](https://ckir.github.io/corelib/api/)**
  - **[Core Utilities Reference](https://ckir.github.io/corelib/api/core/)** (`@ckirg/corelib`)
  - **[Market Data Reference](https://ckir.github.io/corelib/api/markets/)** (`@ckirg/corelib-markets`)
  - **[Cloud Extensions Reference](https://ckir.github.io/corelib/api/cloud/)** (`@ckirg/corelib-cloud`)
  - **[Rust Native Core Reference](https://ckir.github.io/corelib/api/rust/)** (`corelib-rust`; redirects to the crate page)
- **[User Guide](https://ckir.github.io/corelib/guide/)** ([PDF](https://ckir.github.io/corelib/guide/USER_GUIDE.pdf))

The site is authored in `site/`; the generated API reference and the user guide are merged into
`_site/` by `pnpm site:build`. `pnpm site:preview` serves it locally at
`http://127.0.0.1:4173`.

---

## 📜 License

Refer to the [LICENSE](./LICENSE) file for details.