# Social Media Poster — four layers

Research → create & experiment → distribute → measure, and the measurements steer the
next research. Everything is a Node script with one machine-readable JSON line as its last
output, scheduled by cron, sharing one SQLite file. A PHP web UI is the control panel.

```
L1 RESEARCH      pipeline/research.mjs   seeds → Keywords Everywhere → treg (opt.) → scraper → Jev → topics
L2 CREATION      pipeline/loop.mjs       top topic → LLM copy → template × hook variant → HyperFrames → renders
                 pipeline/autoresearch.mjs (A/B arms)  ·  hyperframes/program.md (per-template critic loop)
L3 DISTRIBUTION  pipeline/publish.mjs    Jev gate → drafts on Postiz (IG/YT/TikTok) + Zernio (Pinterest)
                                         → approval: web Queue tab or Discord/email digest → scheduled
L4 ANALYTICS     pipeline/metrics.mjs    publisher analytics (+ YouTube Data API) → metrics → attribution view
                                         → back into L1 seeds, L2 variant weights, autoresearch verdicts
```

State: `pipeline/data/state.db` (SQLite via `node:sqlite`, schema in `pipeline/db.mjs`):
`topics → renders → posts → metrics`, plus the `attribution` view that joins each post's
latest snapshot to the **template_variant × hook_variant × topic** that produced it. That
join is what makes Layer 2's experiments measurable rather than decorative.

## What is running

| Piece | Where | Status |
|---|---|---|
| Web UI | `http://<this-server>/socialmedia` — Research · Composer · Queue · Metrics · Auto-loop · Settings | ✅ |
| Cron | `/etc/cron.d/socialmedia` — 02:00 metrics+autoresearch · 06:00 research · 07:00 produce · 08:00 publish+digest | ✅ |
| HyperFrames | `hyperframes/templates/{hook-bold,quote-card,listicle}` — CLI pinned in `pipeline/hf.mjs` | ✅ renders verified, deterministic |
| Postiz (IG / YouTube / TikTok) | Docker, `127.0.0.1:4007` | ✅ up · ⛔ channels need your dev apps + public HTTPS |
| Zernio (Pinterest) | hosted, zernio.com | ⏳ needs your account + API key |
| Keywords Everywhere · treg · Jev | Layer 1 signal + scoring | ⏳ keys not set — runs on the free path meanwhile |

Without any paid key the loop still works end to end: seeds + YouTube trend terms → heuristic
topic score → LLM copy (OpenRouter) → render → queue. Keys upgrade each step in place.

## Layout

- `pipeline/db.mjs` — the store. `node pipeline/db.mjs "<sql>"` prints rows as JSON.
- `pipeline/config.mjs` — `loadNiche()` (DEFAULTS ← `loop.config.json` ← `niches/<slug>/niche.json`) and `loadSecrets()` (`config.json`).
- `pipeline/research.mjs` · `loop.mjs` · `publish.mjs` · `metrics.mjs` — the four layers (below).
- `pipeline/poster.mjs` — render one job (HyperFrames template) and post it through each platform's publisher. Used by L2 (`--dry-run`), L3, the web UI and the openclaw skill.
- `pipeline/publish/` — publisher adapters `postiz.mjs`, `zernio.mjs`; `index.mjs` routes per platform (`publishers` map).
- `pipeline/autoresearch.mjs` — Karpathy-style hypothesis → A/B → measure → conclude over `renders`+`attribution`. `pipeline/research-state.json` is its state.
- `pipeline/jev.mjs` (TypeSafe System One), `pipeline/treg.mjs` (treg.to gateway), `scraper/keywords.mjs` (Keywords Everywhere), `scraper/scrape.mjs` (free YouTube + Apify/Reddit sources).
- `hyperframes/` — templates + the critic loop (`program.md`, frozen `evaluate.mjs`, `rubric.md`, `DESIGN.md`).
- `niches/<slug>/niche.json` + `exemplars.json` — a niche pack: pillars, seeds, caps, templates, cadence, styles, proven posts.
- `web/` — the PHP UI (the only web-served dir). `renders/` holds the MP4s the Queue and digest preview.
- `openclaw-skill/social-poster/SKILL.md` — the openclaw agent skill. `postiz/` — the Postiz Docker stack.

## Layer 1 — research (`pipeline/research.mjs`, 06:00)

```
node pipeline/research.mjs --config=niches/<slug>/niche.json [--seed="kw"] [--cap=N] [--dry-run]
```

1. **Seeds** = `niche.seeds` ∪ `--seed` ∪ the best-viewed recent topics from `attribution` (Layer 4 → Layer 1), capped at `maxSeedsPerRun`.
2. **Keywords Everywhere** — related keywords per seed, then volume / CPC / competition for the union (≤ 100 keywords = ≤ 100 credits). Key: `scraper/config.json` → `keywordsEverywhereApiKey`.
3. **Scraper** — what's performing in the niche now (free YouTube path; Apify/Reddit sources when their keys exist). Trend terms boost overlapping candidates.
4. **treg** (optional) — Semrush, DataForSEO, SerpApi and the rest of treg's catalog through one `trg_live_` token, pay-per-call. `config.json` → `treg.sources` is a list of `{role, id, params}`:
   - `related` — seed → keyword ideas with volume/cpc/competition (e.g. `semrush.google.keywords.ideas`), added as candidates;
   - `keywords` — keyword → volume/cpc/competition for candidates KE didn't price (e.g. `semrush.google.keywords.volume`);
   - `serp` — keyword → SERP payload for the top candidates, handed to Jev unparsed (Jev interprets it).
   `params` take `{seed}` / `{keyword}` placeholders; providers' field names are normalised (`treg.mjs`). One budget of `tregMaxCalls` per run, each call capped at `treg.maxCostUsd` (`X-Treg-Route-Max-Cost`); calls and spend are reported. Pick ids and their exact param names with `treg catalog search "keyword research"` / `treg catalog get <id>`. Skipped without a token or sources.
5. **Pre-rank** heuristically (volume × competition × trend overlap) and cap at `dailyTopicCap`.
6. **Jev** — one call per topic: short-form potential 0-10, content pillar (`niche.pillars`), evergreen-vs-spike. Without a key the heuristic score stands in (`jev_labels.basis = "heuristic"`).
7. `INSERT OR IGNORE` into `topics` (status `queued`) — reruns never duplicate.

The Research tab lists the queue and has a **Run research** button (`web/topics.php`).

## Layer 2 — creation & experimentation (`pipeline/loop.mjs`, 07:00)

```
node pipeline/loop.mjs --config=niches/<slug>/niche.json --count=N [--no-render] [--topic=ID]
```

Per render: the highest-scored queued topic → **template_variant** (from `niche.templates`) and
**hook_variant** (number / negative / question / curiosity) — the live autoresearch arm when one
applies, otherwise weighted by mean views per variant from Layer 4 (round-robin until there is
data) → the LLM in `model` writes hook / captions / caption under every experiment rule
(OpenRouter or any OpenAI-compatible API; heuristic engine without a key) → rubric self-score
0-100 → `poster.mjs --dry-run` renders the template → ffprobe QC → a `renders` row; the topic
becomes `used`. Output lands in `web/renders/loop-<ts>/`.

**Experimentation** has two loops:
- `pipeline/autoresearch.mjs` (02:00, after metrics) — factors `template`, `hookStyle`, `ctaVerb`, `ctaPlacement`, `captionCount`, `pacing`. One active experiment; every render is assigned an arm; conclusion needs `--min=4` real samples per arm from `attribution`; concluded winners become generation defaults; an empty queue gets new hypotheses proposed by the LLM. `--self-test` checks the analysis.
- `hyperframes/program.md` — the per-template critic loop: `TEMPLATE=quote-card node evaluate.mjs` renders, lints and judges one template into a scalar METRIC (lower is better); edit only that template's `index.html`, commit on improvement.

Templates share one contract: the single-line `window.__props = {…};` that `poster.mjs`
substitutes with `job.video`, and a root `data-duration` = intro + n × secondsPerCaption + outro.
`npx hyperframes@$(node -p "require('./pipeline/hf.mjs')" 2>/dev/null || echo 0.8.58) check --json .` inside a template dir must be error-free.

## Layer 3 — distribution & approval (`pipeline/publish.mjs`, 08:00)

```
node pipeline/publish.mjs --config=niches/<slug>/niche.json [--limit=N] [--dry-run] [--retry]
node pipeline/publish.mjs --digest
node pipeline/publish.mjs --approve=<postId,…> | --approve-render=<id> | --approve-digest=<id>
node pipeline/publish.mjs --reject=<postId,…>  | --reject-render=<id>
```

For every render with no posts: **Jev gate** → `allow` / `confirm` / `block` (`block` → `posts`
rows `rejected` with the reason; nothing is sent). The render is created as a **draft** on each
platform's publisher at the next free slot from `niche.cadence` (`slotsLocal` in `timezone`,
`perDay`), one `posts` row per platform with `provider`, `external_id`, `gate`. With
`autoPublish: true`, gate `allow` and score ≥ `gateThreshold` it is created as `schedule`
directly (`approved`) — the queue is for everything else.

**Approval** — the **Queue** tab (`web/queue.php`) shows each draft with preview, topic, variants,
score, gate verdict, slot and per-platform status, with Approve / Reject per render and
Approve-all. `--digest` posts the same list to a Discord webhook and/or emails it
(`sendmail`), with signed one-click links (`web/approve.php`, HMAC with `digestSecret`).
Approve → `PUT /posts/{id}/status {schedule}` on Postiz, `PUT /v1/posts/{id} {isDraft:false,
scheduledFor}` on Zernio; a draft approved after its slot passed is re-slotted (Postiz can't
move a date, so it is deleted and recreated). Reject → delete on the publisher.

### Publishers — who posts where

`pipeline/config.json` → `"publishers": { "pinterest": "zernio", "*": "postiz" }` (Settings tab
has the same four selects). A job spanning both backends uploads and posts once per backend;
a backend with no key or no connected channel fails only its own platforms (`errors` in the
result / `failed` posts rows — fix, then `publish.mjs --retry`).

**Postiz** (self-hosted, `postiz/`): channels connect only through your own developer apps and a
public HTTPS origin — DNS A `postiz.funy.click → 165.232.141.92`, `certbot --apache -d
postiz.funy.click`, flip `POSTIZ_PUBLIC_URL` in `postiz/.env`, `docker compose up -d`; app
credentials for Meta / Google / TikTok go in `postiz/.env`; connect channels in the dashboard;
Settings → Public API → key → Settings tab. Dashboard from a laptop: `ssh -L 4007:localhost:4007 root@165.232.141.92`.
The analytics (`GET /analytics/post/{id}`) and status (`PUT /posts/{id}/status`) endpoints exist
in current Postiz; if the running image 404s them, `cd postiz && docker compose pull && docker compose up -d`.

**Zernio** (hosted): sign up at zernio.com (no card; first 2 accounts free), connect Pinterest,
Dashboard → API keys → Settings tab. Board id: `GET /v1/accounts/{id}/pinterest-boards`
(numeric — a name is rejected); the "Visit site" link and board are per post
(`settings.pinterest.{link,board}`) with `pinterestLink` / `pinterestBoard` as defaults.
Zernio derives the video pin's cover from the frame at 1 s; Postiz needs the cover uploaded
as a second media item, which `poster.mjs` cuts with ffmpeg.

## Layer 4 — analytics & feedback (`pipeline/metrics.mjs`, 02:00)

```
node pipeline/metrics.mjs [--days=90] [--post=ID]
```

For every `approved` / `published` post whose slot has passed: provider stats (Postiz post
analytics, Zernio `GET /analytics?postId=`), provider state (→ `published`, live URL) and, for
YouTube, the free Data API (`scraper/config.json` → `youtubeApiKey`) filling anything the
provider left null. One `metrics` row per post per run.

Feedback paths, all through the `attribution` view:
- **→ L1**: `research.mjs` seeds from the best-viewed recent topics.
- **→ L2**: `loop.mjs` weights template/hook picks by mean views (15% exploration floor).
- **→ experiments**: `autoresearch.mjs` judges arms on real views/engagement.
- **→ you**: the Metrics tab — per template, per hook style, per pillar, per platform, top topics.

## Web UI

`/socialmedia` — **Research** (multi-source niche research · SEO keywords · topic queue + Run
research), **Composer** (hand-write a post, pick a template, render preview / publish),
**Queue** (approve / reject drafts), **Metrics** (attribution dashboard, pull analytics now),
**Auto-loop** (run a Layer 2 batch, see recent renders), **Settings** (publisher routing and
every key: Postiz, Zernio, YouTube, Apify, Keywords Everywhere, Reddit, LLM, Jev,
treg + endpoint ids, Discord webhook, digest email/secret, public base URL). Keys are stored
server-side in `pipeline/config.json` / `scraper/config.json` and never sent to the browser.

## Keys — where each one lives

| Key | File | Used by |
|---|---|---|
| `publishers`, `apiKey` (Postiz), `postizUrl`, `zernioApiKey`, `pinterestLink`, `pinterestBoard` | `pipeline/config.json` | poster, publish, metrics, health |
| `OPENROUTER_API_KEY` (or `LLM_API_KEY` + `LLM_BASE_URL`) in `/etc/environment` — **used first**; `llmApiKey` / `llmBaseUrl` in `pipeline/config.json` (Settings tab) are the fallback; `model` in `loop.config.json` | env → config | loop (copy), autoresearch (hypotheses), evaluate (judge, `JUDGE_MODEL`, env only) |
| `jevApiKey` | `pipeline/config.json` | research (score), publish (gate) |
| `tregToken`, `treg.sources[]{role,id,params}`, `treg.maxCostUsd` | `pipeline/config.json` | research |
| `discordWebhookUrl`, `digestEmail`, `digestSecret`, `publicBaseUrl` | `pipeline/config.json` | publish --digest, approve.php |
| `keywordsEverywhereApiKey`, `youtubeApiKey`, `apify.*`, `reddit.*` | `scraper/config.json` | research, scraper, metrics |

`OPENROUTER_API_KEY` lives in `/etc/environment`. Cron inherits it through pam_env; Apache gets it
through `/etc/systemd/system/apache2.service.d/environment.conf` (`EnvironmentFile=/etc/environment`),
so UI-triggered renders use the same variable. The Settings tab's LLM key is only consulted when the
environment has none. Both config files are `root:www-data 640` so the UI can read and write them.

## Verify

```
node pipeline/db.mjs                                        # tables + view
node pipeline/research.mjs --config=niches/dark-motivational/niche.json --dry-run
node pipeline/loop.mjs --count=1 --config=niches/dark-motivational/niche.json      # → renders row + web/renders/loop-*/video.mp4
node pipeline/publish.mjs --dry-run --config=niches/dark-motivational/niche.json   # gate verdicts + routing, nothing sent
node pipeline/publish.mjs --digest --dry-run
node pipeline/metrics.mjs && node pipeline/autoresearch.mjs --self-test
# determinism: render the same job twice, compare frame hashes
node pipeline/poster.mjs job.json --dry-run --out=/tmp/a.mp4 && node pipeline/poster.mjs job.json --dry-run --out=/tmp/b.mp4
for f in a b; do ffmpeg -v error -i /tmp/$f.mp4 -map 0:v -f framemd5 - | grep -v '^#' | md5sum; done
```

## Ops notes

- 2 vCPU / 8 GB box shared with Postiz + Temporal + Elasticsearch: keep `--count` small; a render takes ~1.5 min.
- `pipeline/data/` and `pipeline/logs/` are `root:www-data 2775`; the DB file is `0664` so Apache's `www-data` (Queue approve, Run research) can write. SQLite is in WAL mode.
- `web/renders/`, `pipeline/out/`, `pipeline/data/`, `pipeline/logs/`, both config files and `postiz/.env` are gitignored.
- Postiz backend 502 after a restart → `docker restart postiz` (it needs Temporal up first).
- Everything is a **draft** until approved. Nothing goes live unless you approve it in the Queue / digest, or `autoPublish` is on and both gates pass.