Skip to content

Commit b5f7f10

Browse files
sergeytimoshinsergeytimoshin
authored
feat: forester dashboard + compression improvements (#2310)
* fix: update account data handling to strip discriminator prefix and include discriminator length * chore: update photon subproject and refactor account data handling to remove discriminator length * chore: update photon subproject to latest commit * chore: update photon subproject to latest commit * chore: update photon subproject to latest commit * feat: add --helius-rpc cli flag feat: add support for getProgramAccounts standard rpc calls for compression feat: structured error logging * feat: enhance compressible data tracking - Introduced `forester_api_urls` argument in `DashboardArgs` for specifying multiple API base URLs. - Enhanced `EpochManager` to handle non-retryable registration errors gracefully. - Implemented `CompressibleTrackerHandles` struct to manage multiple trackers for compressible data. - Refactored `initialize_compressible_trackers` to streamline tracker initialization and bootstrap processes. - Updated `run_pipeline_with_run_id` to accept preconfigured tracker handles, improving modularity. - Modified `main` function to initialize compressible trackers and manage shutdown signals effectively. * fix: improve transaction handling in compressors - Enhanced error handling in `CTokenCompressor`, `MintCompressor`, and `PdaCompressor` to manage pending states more effectively. - Added checks to ensure accounts are marked as pending during transaction processing. chore: update epoch manager to check eligibility for compression - Modified `dispatch_compression`, `dispatch_pda_compression`, and `dispatch_mint_compression` to include eligibility checks based on the current light slot. refactor: improve account tracking with atomic counters - Introduced `compressed_count` and `pending` sets in account trackers for better management of compression states. - Updated `CompressibleTracker` trait to include methods for managing pending accounts and counting compressed accounts. fix: ensure proper handling of closed accounts in trackers - Added logic to remove closed accounts from trackers in `CTokenAccountTracker`, `MintAccountTracker`, and `PdaAccountTracker`. feat: add usePhotonStats hook for fetching photon statistics - Implemented a new hook `usePhotonStats` using SWR for fetching photon statistics from the API. - Introduced error handling for API responses. refactor: enhance utility functions for address exploration - Added `explorerUrl` function to generate Solana explorer URLs based on the current network. - Improved `formatSlotCountdown` to handle additional parameters for better status reporting. feat: extend forester types with new statistics - Updated `AggregateQueueStats`, `ForesterStatus`, and related interfaces to include new fields for batch processing statistics. - Introduced `PhotonStats` interface for tracking photon-related metrics. * fix: improve error handling and method consistency in compressors and state management * feat: add transaction verification to compressors and refactor MintAccountTracker initialization * fix: improve transaction confirmation handling and error reporting in compressors * fix: remove unnecessary pubkey collection before marking accounts as pending * chore: update subproject commit for photon dependency * fix: implement retry logic for transaction status verification * refactor transaction handling
1 parent a44b61f commit b5f7f10

60 files changed

Lines changed: 4620 additions & 1456 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎external/photon‎

Submodule photon updated 32 files

‎forester/dashboard/package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,14 @@
1010
},
1111
"dependencies": {
1212
"next": "^14.2.0",
13+
"pg": "^8.18.0",
1314
"react": "^18.3.0",
1415
"react-dom": "^18.3.0",
1516
"swr": "^2.2.0"
1617
},
1718
"devDependencies": {
1819
"@types/node": "^20.0.0",
20+
"@types/pg": "^8.16.0",
1921
"@types/react": "^18.3.0",
2022
"@types/react-dom": "^18.3.0",
2123
"autoprefixer": "^10.4.0",

‎forester/dashboard/src/app/api/[...path]/route.ts‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
import { NextResponse } from "next/server";
2+
3+
export const dynamic = "force-dynamic";
4+
5+
const BACKEND_URL =
6+
process.env.FORESTER_API_URL || "http://127.0.0.1:8080";
7+
8+
const BACKEND_TIMEOUT_MS = Number(
9+
process.env.FORESTER_API_TIMEOUT_MS ?? 8000
10+
);
11+
12+
function isAbortError(error: unknown): boolean {
13+
return (
14+
typeof error === "object" &&
15+
error !== null &&
16+
"name" in error &&
17+
(error as { name?: string }).name === "AbortError"
18+
);
19+
}
20+
21+
function joinBackendUrl(path: string): string {
22+
const base = BACKEND_URL.replace(/\/+$/, "");
23+
return `${base}/${path}`;
24+
}
25+
26+
export async function GET(
27+
_request: Request,
28+
{ params }: { params: Promise<{ path: string[] }> }
29+
) {
30+
const { path } = await params;
31+
const backendPath = path.join("/");
32+
const upstream = joinBackendUrl(backendPath);
33+
34+
const timeoutMs =
35+
Number.isFinite(BACKEND_TIMEOUT_MS) && BACKEND_TIMEOUT_MS > 0
36+
? BACKEND_TIMEOUT_MS
37+
: 8000;
38+
39+
const controller = new AbortController();
40+
const timer = setTimeout(() => controller.abort(), timeoutMs);
41+
42+
try {
43+
const response = await fetch(upstream, {
44+
cache: "no-store",
45+
signal: controller.signal,
46+
});
47+
48+
const contentType = response.headers.get("content-type") ?? "";
49+
const payload = contentType.includes("application/json")
50+
? await response.json()
51+
: { message: await response.text() };
52+
53+
if (!response.ok) {
54+
return NextResponse.json(
55+
{
56+
error: `Forester backend returned ${response.status}`,
57+
upstream,
58+
details: payload,
59+
},
60+
{ status: response.status }
61+
);
62+
}
63+
64+
return NextResponse.json(payload, { status: response.status });
65+
} catch (error) {
66+
if (isAbortError(error)) {
67+
return NextResponse.json(
68+
{
69+
error: `Backend request timed out after ${timeoutMs}ms`,
70+
upstream,
71+
},
72+
{ status: 504 }
73+
);
74+
}
75+
76+
return NextResponse.json(
77+
{ error: "Backend unavailable", upstream },
78+
{ status: 502 }
79+
);
80+
} finally {
81+
clearTimeout(timer);
82+
}
83+
}

‎forester/dashboard/src/app/api/photon-stats/route.ts‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
import { NextResponse } from "next/server";
2+
import { Pool } from "pg";
3+
4+
export const dynamic = "force-dynamic";
5+
6+
let pool: Pool | null = null;
7+
8+
function getPool(): Pool | null {
9+
const url = process.env.PHOTON_DATABASE_URL;
10+
if (!url) return null;
11+
if (!pool) {
12+
pool = new Pool({ connectionString: url, max: 2, idleTimeoutMillis: 30_000 });
13+
}
14+
return pool;
15+
}
16+
17+
export async function GET() {
18+
const db = getPool();
19+
if (!db) {
20+
return NextResponse.json(
21+
{ error: "PHOTON_DATABASE_URL not configured" },
22+
{ status: 503 }
23+
);
24+
}
25+
26+
try {
27+
const client = await db.connect();
28+
try {
29+
const [accounts, tokens, compressed] = await Promise.all([
30+
client.query(`
31+
SELECT
32+
COUNT(*) AS total,
33+
COUNT(*) FILTER (WHERE NOT spent) AS active
34+
FROM accounts
35+
`),
36+
client.query(`
37+
SELECT
38+
COUNT(*) AS total,
39+
COUNT(*) FILTER (WHERE NOT spent) AS active
40+
FROM token_accounts
41+
`),
42+
client.query(`
43+
SELECT
44+
encode(a.owner, 'base64') AS owner_b64,
45+
COUNT(*) AS total,
46+
COUNT(*) FILTER (WHERE NOT a.spent) AS active
47+
FROM accounts a
48+
WHERE a.onchain_pubkey IS NOT NULL
49+
GROUP BY a.owner
50+
ORDER BY COUNT(*) DESC
51+
`),
52+
]);
53+
54+
const totalAccounts = Number(accounts.rows[0].total);
55+
const activeAccounts = Number(accounts.rows[0].active);
56+
const totalTokens = Number(tokens.rows[0].total);
57+
const activeTokens = Number(tokens.rows[0].active);
58+
59+
const compressedFromOnchain = compressed.rows.map((r: { owner_b64: string; total: string; active: string }) => ({
60+
owner: Buffer.from(r.owner_b64, "base64").toString("hex"),
61+
total: Number(r.total),
62+
active: Number(r.active),
63+
}));
64+
65+
const totalCompressedFromOnchain = compressedFromOnchain.reduce(
66+
(sum: number, r: { total: number }) => sum + r.total,
67+
0
68+
);
69+
const activeCompressedFromOnchain = compressedFromOnchain.reduce(
70+
(sum: number, r: { active: number }) => sum + r.active,
71+
0
72+
);
73+
74+
return NextResponse.json({
75+
accounts: { total: totalAccounts, active: activeAccounts },
76+
token_accounts: { total: totalTokens, active: activeTokens },
77+
compressed_from_onchain: {
78+
total: totalCompressedFromOnchain,
79+
active: activeCompressedFromOnchain,
80+
by_owner: compressedFromOnchain,
81+
},
82+
timestamp: Math.floor(Date.now() / 1000),
83+
});
84+
} finally {
85+
client.release();
86+
}
87+
} catch (err) {
88+
const message = err instanceof Error ? err.message : String(err);
89+
return NextResponse.json({ error: message }, { status: 500 });
90+
}
91+
}

‎forester/dashboard/src/app/compressible/page.tsx‎

Lines changed: 0 additions & 30 deletions
This file was deleted.

‎forester/dashboard/src/app/layout.tsx‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
import type { Metadata } from "next";
2-
import { Sidebar } from "@/components/Sidebar";
32
import "./globals.css";
43

54
export const metadata: Metadata = {
@@ -14,11 +13,8 @@ export default function RootLayout({
1413
}) {
1514
return (
1615
<html lang="en">
17-
<body className="bg-white text-gray-900 antialiased">
18-
<div className="flex min-h-screen">
19-
<Sidebar />
20-
<main className="flex-1 p-6 overflow-auto">{children}</main>
21-
</div>
16+
<body className="bg-gray-50 text-gray-900 antialiased">
17+
<main className="max-w-6xl mx-auto px-6 py-8">{children}</main>
2218
</body>
2319
</html>
2420
);

‎forester/dashboard/src/app/metrics/page.tsx‎

Lines changed: 0 additions & 44 deletions
This file was deleted.

0 commit comments

Comments
 (0)