Mastodon Cross-Posting Pipeline

mastodon-x-poster.md

Building a Mastodon Cross-Posting Pipeline

This document outlines the technical implementation details for building a real-time, self-hosted Mastodon cross-posting system. It is designed for senior developers who want to replicate this architecture on their own Mastodon instances.

Assumptions & Environment Layout

This guide assumes you are deploying alongside an existing Mastodon installation with the following layout:

• User: Scripts run under the mastodon user.
• Directory: Scripts are located in ~/live/mods/ (or similar directory accessible to the mastodon user).
• Process Management: Systemd user services are available and lingering is enabled (loginctl enable-linger mastodon).
• Runtime: Node.js (ESM modules enabled). Note: The reference implementation was constrained to Node v16.20.1, requiring polyfills for fetch via undici and node-fetch, and older import assertion syntax (assert { type: "json" }). Modern Node (18+) simplifies this.

Core Architectural Concepts

1. Mastodon as the Single Source of Truth (SSOT)

All content originates on Mastodon. The system does not use a separate database (like PostgreSQL or Redis) to track what has been cross-posted. Instead, it uses Mastodon itself as the state store.

2. The "Hidden Reply" State Store

To track cross-posting metadata (e.g., the resulting Tweet URL, Nostr event ID, Bluesky URI), the system replies to the original Mastodon post with a direct (private) message.

• Visibility: Set to direct. This ensures the reply is only visible to mentioned users (which is just the author). It does not clutter the public timeline.
• Linking: Uses in_reply_to_id pointing to the original post.
• Format: The reply contains key-value pairs and a specific hashtag (e.g., #xpostmeta) to easily identify it among other replies.
• Idempotency: Before posting to a platform, the script checks this hidden reply. If a URL for that platform exists, it skips posting.

3. Local File Caching

To minimize API calls to the local Mastodon instance, the system caches the user's timeline and lookup data in local JSON files (lookup-[user].json, [id]-statuses.json).

4. Sequential Orchestration

Cross-posting happens sequentially (e.g., Nostr -> Twitter -> Bluesky). After each successful post, the metadata reply is updated, and the local Mastodon timeline cache is re-fetched. This prevents race conditions where two scripts try to create/update the metadata reply simultaneously.

---

Component Implementation Details

1. The Real-Time Listener (masto-listener.mjs)

Instead of polling via cron, use Mastodon's Streaming API (WebSocket) for instant cross-posting.

• Endpoint: wss://[server]/api/v1/streaming?access_token=[token]&stream=public:local
• Logic: Listen for update events. Parse the payload, check if status.account.id matches your user ID, and if so, spawn the orchestrator script.
• Resilience: Wrap the WebSocket connection in a systemd service with Restart=always and RestartSec=10 to handle network drops.

2. The Orchestrator (process-incoming-mastodon.sh)

A bash script that acts as the glue.

• Flow:
  1. Run fetch-masto.mjs to update local JSON caches.
  2. If new posts exist (exit code 0), run post-nostr.mjs.
  3. Run fetch-masto.mjs again (to pull the newly created Nostr metadata reply).
  4. Run post-twitter.mjs.
  5. Run fetch-masto.mjs again.
  6. Run post-bluesky.mjs.

3. State Fetcher (fetch-masto.mjs)

Fetches the latest statuses.

• Endpoint: /api/v1/accounts/[id]/statuses?exclude_replies=false&limit=40
• Crucial Detail: exclude_replies=false is mandatory. Otherwise, you won't fetch the hidden #xpostmeta replies needed to determine state.
• Diffing: Compare the created_at timestamp of the latest fetched post against the first item in the previously cached JSON file to determine if a new post exists.

4. Metadata Manager (metadata.mjs)

Handles reading and writing the hidden state replies.

• Parsing: Mastodon returns content as HTML. You must strip tags and decode HTML entities (using a library like he) before parsing the key-value pairs.
• Upsert Logic:
  • Search the cached timeline for a post where in_reply_to_id matches the target post AND the decoded text contains #xpostmeta.
  • If found, issue a PUT request to /api/v1/statuses/[reply_id] with the merged metadata.
  • If not found, issue a POST request to /api/v1/statuses with visibility: 'direct' and in_reply_to_id.

5. Media Caching (media-cache.mjs)

Downloading media multiple times for each platform is inefficient.

• Implementation: Create a /tmp/mastodon-media-cache directory.
• Hashing: Hash the Mastodon media URL (SHA-256) to generate the local filename.
• Mime Types: Ensure you map file extensions correctly, as platforms like Twitter are strict about mime types during media uploads.

---

Platform-Specific Implementation Details

Each cross-posting script follows a similar high-level pattern but handles the specific API quirks, media upload requirements, and text formatting rules of its target platform.

1. Bluesky (post-bluesky.mjs)

Configuration / Environment Variables:

• BLUESKY_HANDLE (e.g., your.handle.bsky.social)
• BLUESKY_APP_PASSWORD

Text Processing & Limits:

• Bluesky has a strict 300-character limit.
• Links and mentions must be explicitly defined using "facets" with exact byte-offsets (UTF-8).
• Because of the character limit, the script manually shortens URLs in the text (e.g., example.com/very/long/path...) and adjusts the facet byte-offsets accordingly.

Media Handling:

• Images: Up to 4 images. Uploaded via agent.uploadBlob(). Requires aspect ratio metadata for optimal display.
• Video: Up to 1 video. Uploaded via agent.uploadBlob(). The script must wait a few seconds after upload for Bluesky's backend to process the video before attaching it to the post.
• Link Cards: If no media is present but a link is, Bluesky requires you to manually construct the link card (app.bsky.embed.external). The script includes a fallback to scrape YouTube's oEmbed API if Mastodon didn't provide a parsed link card.

Pseudocode Outline:

// 1. Check idempotency
if (metadata.Bluesky) exit;

// 2. Format text and determine media strategy
let embed = null;
if (hasLinkCard && !hasMedia) {
  // Upload thumbnail blob, construct external embed
  embed = { $type: 'app.bsky.embed.external', external: { ... } };
} else if (hasVideo) {
  // Upload video blob, wait for processing
  let blob = await agent.uploadBlob(videoBuffer);
  embed = { $type: 'app.bsky.embed.video', video: blob };
} else if (hasImages) {
  // Upload up to 4 image blobs
  embed = { $type: 'app.bsky.embed.images', images: [...] };
}

// 3. Process text facets (links/mentions)
let rt = new RichText({ text: postContent });
await rt.detectFacets(agent);
shortenUrlsAndAdjustFacets(rt);
truncateTo300Characters(rt);

// 4. Post and update state
const res = await agent.post({ text: rt.text, facets: rt.facets, embed });
await upsertMetadata(post, { 'Bluesky': res.uri });

2. Twitter / X (post-twitter.mjs)

Configuration / Environment Variables:

• TWITTER_API_KEY
• TWITTER_API_SECRET
• TWITTER_ACCESS_TOKEN
• TWITTER_ACCESS_SECRET (Note: Requires a "Pay per use" / Basic tier app in the Twitter Developer Portal. Free tier will return 503 errors for media uploads).

Text Processing & Limits:

• Twitter uses a weighted character count (280 limit). URLs always count as 23 characters, regardless of actual length.
• The script uses the official twitter-text library to accurately parse the tweet, calculate the weighted length, and safely truncate without breaking URLs.

Media Handling:

• Two-Step Upload: Media must be uploaded using the v1.1 API (client.v1.uploadMedia) to get a media_id, which is then attached to the v2 API tweet payload.
• Alt Text: Requires a separate v1.1 API call (client.v1.createMediaMetadata) after the image is uploaded.
• Mixed Media Constraint: Twitter's API does not allow mixing images and videos in a single tweet. If a Mastodon post has both, the script uploads the images, but for the video, it uploads the video preview thumbnail (image) instead of the video itself, and appends the video URL to the text.

Pseudocode Outline:

// 1. Check idempotency
if (metadata.Tweet) exit;

// 2. Upload media via v1 API
let mediaIds = [];
if (hasImages && hasVideo) {
  // Twitter can't mix media types. Upload images + video preview thumbnail.
  mediaIds.push(await uploadMedia(image1));
  mediaIds.push(await uploadMedia(videoPreviewThumbnail));
} else if (hasVideo) {
  mediaIds.push(await uploadMedia(video, 'video/mp4'));
} else if (hasImages) {
  for (let img of images) mediaIds.push(await uploadMedia(img));
}

// 3. Format and truncate text
let tweetContent = convertHtmlToText(post.content);
if (!twitterText.parseTweet(tweetContent).valid) {
  tweetContent = safeTruncate(tweetContent, 280);
}

// 4. Post via v2 API and update state
const tweet = await client.v2.tweet({ 
  text: tweetContent, 
  media: mediaIds.length ? { media_ids: mediaIds } : undefined 
});
await upsertMetadata(post, { 'Tweet': `https://twitter.com/i/web/status/${tweet.data.id}` });

3. Nostr (post-nostr.mjs)

Configuration / Environment Variables:

• NOSTR_ID (Public key / npub)
• NOSTR_NSEC (Secret key for signing events)

Text Processing & Limits:

• Nostr clients generally expect plain text. HTML is stripped.
• Media URLs are simply appended to the end of the text content.

Media Handling:

• Nostr does not host media. Media remains hosted on the Mastodon server.
• The script implements NIP-92 (Media Attachments) by adding imeta tags to the event, containing the URL and alt text of the Mastodon-hosted media.

Advanced Idempotency (The masto-id tag): Because Nostr is decentralized, the script uses a highly robust idempotency check. It tags every Nostr event with ["masto-id", mastodon_post_id]. If the local Mastodon metadata reply is ever lost or deleted, the script queries the Nostr relays for events authored by the user containing that masto-id tag. If found, it recovers the state without double-posting.

Pseudocode Outline:

// 1. Local idempotency check
if (metadata.Nostr) exit;

// 2. Construct Event Template
const eventTemplate = {
  kind: 1,
  created_at: post.created_at,
  content: convertHtmlToText(post.content) + "\n" + mediaUrls.join("\n"),
  tags: [
    ["masto-id", post.id], // Crucial for recovery
    ["masto-url", post.url],
    ...media.map(m => ["imeta", `url ${m.url}`, `alt ${m.alt}`])
  ]
};

// 3. Network idempotency check (Recovery)
const relays = await fetchNip65RelayList(NOSTR_ID);
const existingEvents = await pool.querySync(relays, { authors: [NOSTR_ID] });
const alreadyPosted = existingEvents.find(e => e.tags.includes(["masto-id", post.id]));

if (alreadyPosted) {
  // State was lost locally, but exists on network. Just heal local state.
  await upsertMetadata(post, { 'Nostr': neventEncode(alreadyPosted.id) });
  exit;
}

// 4. Sign and Publish
const signedEvent = finalizeEvent(eventTemplate, NOSTR_NSEC);
await pool.publish(relays, signedEvent);
await upsertMetadata(post, { 'Nostr': neventEncode(signedEvent.id) });

---

Anticipated Questions / FAQ

Q: Why not use a SQLite database instead of Mastodon direct replies? A: Using Mastodon as the state store makes the system entirely stateless from a deployment perspective. If the server crashes or the /tmp directory is wiped, the system recovers perfectly because the source of truth (what has been posted where) lives attached to the posts themselves. It also allows the user to manually inspect cross-post links directly in their Mastodon UI.

Q: How are edits and deletions handled? A: This specific architecture is append-only. It filters for new posts and ignores edits/deletes. Implementing edits requires listening for status.update events on the WebSocket, mapping the Mastodon ID to the downstream IDs via the metadata reply, and issuing update/delete API calls to the respective platforms.

Q: What about rate limits? A: Because this system is triggered by a single user's actions via a WebSocket listener, it naturally operates well within the rate limits of all target platforms. The only risk is the initial Mastodon timeline fetch, which is why local JSON caching is used.

Q: Why does the orchestrator script run sequentially instead of in parallel? A: If Nostr, Twitter, and Bluesky scripts run simultaneously, they will all try to create the initial #xpostmeta direct reply on Mastodon at the same time, resulting in duplicate metadata replies. Running sequentially and re-fetching the timeline ensures they append to the same single reply.