Stop failing sync over already-sent campaigns still in the repo

A campaign.md is meant to stay around after it's sent as a record of what
went out and when. Treating any non-draft campaign as a hard rejection
forced deleting it just to keep CI green, and there was no way to fail on
resolving a stale list/attachment reference either -- an archived
campaign has nothing left to resolve, so check status before doing any of
that work. Non-draft campaigns now report as a distinct, non-error
"skipped" bucket instead of "rejected".

README.md

@@ -54,8 +54,11 @@ attachments:
 ---
 
 Campaign body goes here, in Markdown. It's sent to listmonk with
-content_type "markdown" — listmonk renders the HTML *and* derives the
+content_type "markdown" for the HTML part; listmonk never derives a
-plaintext alternative itself.
+plaintext alternative on its own, so this same source is also sent
+verbatim as `altbody` — recipients on plaintext-only clients see the
+raw Markdown (and personalization still resolves, since the `{{ }}`
+expressions are still there), rather than getting an HTML-only email.
 ```
 
 - **`subject`** — required.
@@ -68,6 +71,18 @@ plaintext alternative itself.
 - **`type`** — defaults to `regular`. `optin` isn't supported and is rejected at sync time.
 - **`scheduled_at` doesn't exist** — deliberately, since listmonk auto-sends a `scheduled` campaign the moment it fires, which would let a plain `git push` cause a real send.
 
+## Personalization
+
+Listmonk renders each recipient's campaign body as a Go template before converting Markdown to HTML, so a subscriber's `attribs` can be referenced directly in `campaign.md`'s body. `eec`'s signup forms set `first_name` (and `last_name`, if given) as attribs on every subscriber who provides one — subscribers who signed up before that (or via the public form when it collected email only) won't have it set.
+
+Verified working syntax, with a fallback for subscribers who don't have `first_name` set:
+
+```
+{{ if .Subscriber.Attribs.first_name }}Hi {{ .Subscriber.Attribs.first_name }},{{ else }}Hi there,{{ end }}
+```
+
+**Don't trust listmonk's `/api/campaigns/:id/preview` endpoint (or its admin-UI "Preview" button) to test this** — it always renders against listmonk's built-in demo subscriber (`demo@listmonk.app`, attribs `{city: Bengaluru}`), ignoring which subscriber you're looking at, so `first_name` will always look unset there even when it's really set on a real subscriber. Use `campaigns test SLUG EMAIL...` (a real test-send to a real subscriber's own address) to verify personalization actually renders — that's how the syntax above was confirmed.
+
 ## Segmentation
 
 `segment_query` is a raw Postgres-style SQL boolean expression — the same mechanism listmonk's own admin UI search box accepts. Since a campaign can only target whole list(s), `sync` materializes the query into a managed list:
@@ -86,11 +101,14 @@ Each file in `attachments:` is uploaded to listmonk's media library under a synt
 
 Every `sync` that actually creates or changes a campaign auto-sends a preview via listmonk's test-send endpoint, to `preview_emails` if set, otherwise `CAMPAIGNS_PREVIEW_EMAIL`. `campaigns test` is for an on-demand re-preview without touching content.
 
+## Sent campaigns stay in the repo
+
+A campaign's `campaign.md` is meant to stick around after it's sent — a record of what went out and when, not something to delete once it's live. Once a campaign is anything other than `draft` in listmonk (`scheduled`/`running`/`paused`/`cancelled`/`finished`), sync leaves it alone entirely — no write, no re-preview, and (as of v0.1.6) no failure either. It's reported as **skipped**, a distinct, non-error bucket from rejected.
+
 ## Sync guardrails
 
 A rejected campaign is usually one of these, all deliberate:
 
-- **Anything other than `draft` in listmonk** (`scheduled`/`running`/`paused`/`cancelled`/`finished`) — sync refuses to touch it once it's live or sent.
 - **A `lists:` name matches zero or more than one listmonk list** — sync never guesses.
 - **An invalid `segment_query`** — listmonk's own error is surfaced verbatim.
 - **Missing `subject`/`from_email`/a target audience, or `type: optin`.**
@@ -107,8 +125,8 @@ A starting template (listmonk's own stock `default.tpl`, with the required `{{ U
 
 Each domain/client gets its own content repo, isolated by ordinary Gitea repo permissions — a client with push access to their repo has no path to any other domain's listmonk credentials, because those credentials simply don't exist in their repo.
 
-1. Create a new Gitea repo (e.g. `<client>-campaigns`), private.
+1. Generate a new repo from [`campaigns-template`](https://gitea.reground.org/will/campaigns-template) (Gitea's "Generate Repo," not a clone/fork — it starts the new repo with its own clean history), private.
-2. Add: an empty `campaigns/` directory, `email-templates/campaign.html` (copy the stock template from an existing content repo, or listmonk's own `static/email-templates/default.tpl`), and `.gitea/workflows/{sync,send,test,sync-template}.yml` copied from an existing content repo — each does `go install gitea.reground.org/will/eec-campaigns@<pinned-tag>` then runs the corresponding subcommand, with that repo's own secrets.
+2. Follow that generated repo's README checklist: rename the default-template name, restyle `email-templates/campaign.html` if needed, add secrets, confirm the tool version pin, run `Sync default campaign template` once, replace the README with a short domain-specific one.
 3. Add repo secrets: `CAMPAIGNS_LISTMONK_BASE_URL`, `CAMPAIGNS_LISTMONK_API_USER`, `CAMPAIGNS_LISTMONK_API_TOKEN`, and optionally `CAMPAIGNS_PREVIEW_EMAIL` — scoped to that domain's own listmonk instance and API user.
 4. Manually run the `Sync default campaign template` workflow once before the first real send.
 

internal/campaign/send.go

@@ -39,5 +39,5 @@ func Test(lm *listmonk.Client, slug string, emails []string) error {
 	if camp == nil {
 		return fmt.Errorf("no campaign named %q in listmonk — run sync first", slug)
 	}
-	return lm.TestCampaign(camp.ID, emails)
+	return lm.TestCampaign(camp.ID, camp.AsInput(), emails)
 }

internal/campaign/sync.go

@@ -18,7 +18,8 @@ import (
 type SyncResult struct {
 	Synced        []string // created or updated (content changed) in listmonk
 	Unchanged     []string // existing draft, content identical — no API write, no preview
-	Rejected      []string // validation/segmentation/list-lookup failure, or non-draft in listmonk — nothing written
+	Skipped       []string // already sent/running/paused/etc. in listmonk — a historical record, left untouched, not an error
+	Rejected      []string // validation/segmentation/list-lookup failure — nothing written
 	PreviewFailed []string // content synced fine, but the automatic preview send itself failed
 }
 
@@ -45,6 +46,10 @@ func SyncDir(lm *listmonk.Client, root string, defaultPreviewEmails []string) (*
 			result.Rejected = append(result.Rejected, fmt.Sprintf("%s: %v", slug, err))
 			continue
 		}
+		if res.Skipped != "" {
+			result.Skipped = append(result.Skipped, fmt.Sprintf("%s: %s", slug, res.Skipped))
+			continue
+		}
 		if !res.Changed {
 			result.Unchanged = append(result.Unchanged, slug)
 			continue
@@ -59,6 +64,7 @@ func SyncDir(lm *listmonk.Client, root string, defaultPreviewEmails []string) (*
 
 type syncOneResult struct {
 	Changed    bool
+	Skipped    string // non-empty (the listmonk status) when left untouched because it's no longer a draft
 	PreviewErr error  // set only when Changed and the automatic preview send failed
 }
 
@@ -68,6 +74,22 @@ func syncOne(lm *listmonk.Client, slug, dir, path string, defaultPreviewEmails [
 		return syncOneResult{}, err
 	}
 
+	existing, err := lm.FindCampaignByName(slug)
+	if err != nil {
+		return syncOneResult{}, err
+	}
+	if existing != nil && existing.Status != "draft" {
+		// Once a campaign has actually gone out (or is running/paused/etc.),
+		// its campaign.md is a historical record of what was sent and when —
+		// not something sync should touch or fail the build over. Forcing it
+		// to be deleted just to keep CI green would throw away exactly the
+		// record someone would want to look back at later. Check this before
+		// resolving anything else below, so a since-renamed list or a
+		// since-changed attachment on an old campaign can never break the
+		// build either — an archived campaign has nothing left to resolve.
+		return syncOneResult{Skipped: existing.Status}, nil
+	}
+
 	listIDs, err := resolveLists(lm, fm.Lists)
 	if err != nil {
 		return syncOneResult{}, err
@@ -97,11 +119,6 @@ func syncOne(lm *listmonk.Client, slug, dir, path string, defaultPreviewEmails [
 		MediaIDs:   mediaIDs,
 	}
 
-	existing, err := lm.FindCampaignByName(slug)
-	if err != nil {
-		return syncOneResult{}, err
-	}
-
 	var campaignID int
 	var changed bool
 	switch {
@@ -112,11 +129,6 @@ func syncOne(lm *listmonk.Client, slug, dir, path string, defaultPreviewEmails [
 		}
 		campaignID, changed = created.ID, true
 
-	case existing.Status != "draft":
-		// The core safety invariant: never silently skip or overwrite a
-		// live/sent campaign. This is a hard rejection, not a warning.
-		return syncOneResult{}, fmt.Errorf("campaign is %s in listmonk; sync refuses to modify a non-draft campaign", existing.Status)
-
 	case campaignUnchanged(existing, input):
 		campaignID, changed = existing.ID, false
 
@@ -137,7 +149,7 @@ func syncOne(lm *listmonk.Client, slug, dir, path string, defaultPreviewEmails [
 	}
 	var previewErr error
 	if len(recipients) > 0 {
-		previewErr = lm.TestCampaign(campaignID, recipients)
+		previewErr = lm.TestCampaign(campaignID, input, recipients)
 	}
 	return syncOneResult{Changed: true, PreviewErr: previewErr}, nil
 }
@@ -217,8 +229,18 @@ func attachmentFilename(slug, relPath string, content []byte) string {
 func campaignUnchanged(existing *listmonk.Campaign, input listmonk.CampaignInput) bool {
 	return existing.Subject == input.Subject &&
 		existing.Body == input.Body &&
+		// altbody always mirrors body (see CampaignInput.payload) -- comparing
+		// it here is what catches a campaign synced before that started, whose
+		// stored altbody is still null even though its body hasn't changed.
+		existing.AltBody == input.Body &&
 		existing.FromEmail == input.FromEmail &&
-		existing.TemplateID == input.TemplateID &&
+		// input.TemplateID == 0 means "unspecified, use whatever's assigned"
+		// (see CampaignInput.TemplateID) -- listmonk always assigns some real,
+		// nonzero template_id server-side even when it's omitted from the
+		// create/update payload, so comparing 0 against that would flag every
+		// campaign.md without an explicit template_id as changed on every
+		// sync, forever.
+		(input.TemplateID == 0 || existing.TemplateID == input.TemplateID) &&
 		existing.Type == input.Type &&
 		equalStringSets(existing.Tags, input.Tags) &&
 		equalIntSets(existing.ListIDs, input.ListIDs) &&

internal/campaign/sync_test.go

@@ -31,6 +31,7 @@ type fakeCampaign struct {
 	Status     string
 	Subject    string
 	Body       string
+	AltBody    string
 	FromEmail  string
 	TemplateID int
 	Type       string
@@ -138,7 +139,7 @@ func (c fakeCampaign) toJSON() map[string]any {
 	}
 	return map[string]any{
 		"id": c.ID, "name": c.Name, "status": c.Status, "subject": c.Subject,
-		"body": c.Body, "from_email": c.FromEmail, "template_id": c.TemplateID,
+		"body": c.Body, "altbody": c.AltBody, "from_email": c.FromEmail, "template_id": c.TemplateID,
 		"type": c.Type, "tags": c.Tags, "lists": lists, "media": media,
 	}
 }
@@ -151,12 +152,25 @@ func (f *fakeListmonk) writeCampaigns(w http.ResponseWriter) {
 	json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"results": results}})
 }
 
+// fakeDefaultTemplateID is the nonzero template_id real listmonk assigns
+// server-side to a campaign whose create/update request omitted template_id
+// entirely -- never 0, which is what makes the zero value ambiguous between
+// "unspecified" and "really is 0" in campaignUnchanged.
+const fakeDefaultTemplateID = 1
+
+func (f *fakeListmonk) resolveTemplateID(body map[string]any) int {
+	if v, ok := body["template_id"]; ok {
+		return toInt(v)
+	}
+	return fakeDefaultTemplateID
+}
+
 func (f *fakeListmonk) createCampaign(w http.ResponseWriter, r *http.Request) {
 	body := decodeBody(r)
 	c := fakeCampaign{
 		ID: f.id(), Name: str(body["name"]), Status: "draft",
-		Subject: str(body["subject"]), Body: str(body["body"]),
+		Subject: str(body["subject"]), Body: str(body["body"]), AltBody: str(body["altbody"]),
-		FromEmail: str(body["from_email"]), TemplateID: toInt(body["template_id"]),
+		FromEmail: str(body["from_email"]), TemplateID: f.resolveTemplateID(body),
 		Type: str(body["type"]), Tags: toStrings(body["tags"]),
 		ListIDs: toInts(body["lists"]), MediaIDs: toInts(body["media"]),
 	}
@@ -171,8 +185,9 @@ func (f *fakeListmonk) updateCampaign(w http.ResponseWriter, r *http.Request) {
 		if f.campaigns[i].ID == id {
 			f.campaigns[i].Subject = str(body["subject"])
 			f.campaigns[i].Body = str(body["body"])
+			f.campaigns[i].AltBody = str(body["altbody"])
 			f.campaigns[i].FromEmail = str(body["from_email"])
-			f.campaigns[i].TemplateID = toInt(body["template_id"])
+			f.campaigns[i].TemplateID = f.resolveTemplateID(body)
 			f.campaigns[i].Type = str(body["type"])
 			f.campaigns[i].Tags = toStrings(body["tags"])
 			f.campaigns[i].ListIDs = toInts(body["lists"])
@@ -407,6 +422,39 @@ func TestSyncDir_SecondSyncWithUnchangedContentIsNoop(t *testing.T) {
 	}
 }
 
+func TestSyncDir_UnspecifiedTemplateIDDoesNotTriggerSpuriousResync(t *testing.T) {
+	f := newFakeListmonk(t)
+	f.lists = []fakeList{{ID: 3, Name: "Newsletter"}}
+	lm := f.client()
+
+	root := t.TempDir()
+	// baseFrontmatter never sets template_id, matching real campaign.md files
+	// like reground-campaigns' welcome/campaign.md.
+	writeCampaignDir(t, root, "launch", baseFrontmatter, "Hello world.")
+
+	if _, err := SyncDir(lm, root, nil); err != nil {
+		t.Fatalf("first SyncDir: %v", err)
+	}
+	if f.campaigns[0].TemplateID != fakeDefaultTemplateID {
+		t.Fatalf("expected the fake to assign its default template_id like real listmonk does, got %d", f.campaigns[0].TemplateID)
+	}
+
+	// Re-sync with the exact same, still-unspecified template_id: this must
+	// not be flagged as changed just because listmonk's stored, server-
+	// assigned template_id (nonzero) doesn't literally equal the frontmatter's
+	// unset value (0).
+	result, err := SyncDir(lm, root, nil)
+	if err != nil {
+		t.Fatalf("second SyncDir: %v", err)
+	}
+	if len(result.Synced) != 0 {
+		t.Errorf("expected no spurious re-sync from an unspecified template_id, got Synced=%v", result.Synced)
+	}
+	if len(result.Unchanged) != 1 || result.Unchanged[0] != "launch" {
+		t.Errorf("expected launch in Unchanged, got %v", result.Unchanged)
+	}
+}
+
 func TestSyncDir_UpdatesWhenContentChanges(t *testing.T) {
 	f := newFakeListmonk(t)
 	f.lists = []fakeList{{ID: 3, Name: "Newsletter"}}
@@ -431,7 +479,7 @@ func TestSyncDir_UpdatesWhenContentChanges(t *testing.T) {
 	}
 }
 
-func TestSyncDir_RejectsNonDraftCampaign(t *testing.T) {
+func TestSyncDir_SkipsNonDraftCampaignWithoutFailing(t *testing.T) {
 	f := newFakeListmonk(t)
 	f.lists = []fakeList{{ID: 3, Name: "Newsletter"}}
 	f.campaigns = []fakeCampaign{{ID: 1, Name: "launch", Status: "running", Subject: "Original"}}
@@ -444,14 +492,44 @@ func TestSyncDir_RejectsNonDraftCampaign(t *testing.T) {
 	if err != nil {
 		t.Fatalf("SyncDir: %v", err)
 	}
-	if len(result.Rejected) != 1 || !strings.Contains(result.Rejected[0], "running") {
+	// A sent/running campaign is a historical record, not an error -- it
-		t.Fatalf("expected a rejection mentioning 'running', got %v", result.Rejected)
+	// must never land in Rejected (which fails the whole CI job) just
+	// because its campaign.md is still around.
+	if len(result.Rejected) != 0 {
+		t.Fatalf("expected no rejection for an already-sent campaign, got %v", result.Rejected)
+	}
+	if len(result.Skipped) != 1 || !strings.Contains(result.Skipped[0], "running") {
+		t.Fatalf("expected launch in Skipped mentioning 'running', got %v", result.Skipped)
 	}
 	if f.campaigns[0].Subject != "Original" {
 		t.Errorf("expected the live campaign's content to be untouched, got %+v", f.campaigns[0])
 	}
 }
 
+func TestSyncDir_NonDraftCampaignSkipsEvenWithAStaleListReference(t *testing.T) {
+	// A finished campaign's campaign.md might reference a list that's since
+	// been renamed or removed -- that must never break the build, since an
+	// archived campaign has nothing left to resolve.
+	f := newFakeListmonk(t)
+	f.campaigns = []fakeCampaign{{ID: 1, Name: "launch", Status: "finished", Subject: "Original"}}
+	lm := f.client()
+
+	root := t.TempDir()
+	fm := "subject: \"Big Announcement\"\nlists: [\"No Longer Exists\"]\nfrom_email: hello@example.com\n"
+	writeCampaignDir(t, root, "launch", fm, "Hello world.")
+
+	result, err := SyncDir(lm, root, nil)
+	if err != nil {
+		t.Fatalf("SyncDir: %v", err)
+	}
+	if len(result.Rejected) != 0 {
+		t.Fatalf("expected no rejection despite the stale list reference, got %v", result.Rejected)
+	}
+	if len(result.Skipped) != 1 {
+		t.Fatalf("expected launch in Skipped, got %v", result.Skipped)
+	}
+}
+
 func TestSyncDir_AmbiguousListNameRejects(t *testing.T) {
 	f := newFakeListmonk(t)
 	f.lists = []fakeList{{ID: 3, Name: "Newsletter"}, {ID: 4, Name: "Newsletter"}}

internal/listmonk/listmonk.go

@@ -289,6 +289,7 @@ type Campaign struct {
 	Status     string
 	Subject    string
 	Body       string
+	AltBody    string
 	FromEmail  string
 	TemplateID int
 	Type       string
@@ -297,6 +298,23 @@ type Campaign struct {
 	MediaIDs   []int
 }
 
+// AsInput converts a fetched Campaign back into the CampaignInput shape,
+// for callers (e.g. Test) that need to re-send a campaign's current fields
+// against an endpoint that validates the full campaign body.
+func (camp *Campaign) AsInput() CampaignInput {
+	return CampaignInput{
+		Name:       camp.Name,
+		Subject:    camp.Subject,
+		FromEmail:  camp.FromEmail,
+		TemplateID: camp.TemplateID,
+		Type:       camp.Type,
+		Tags:       camp.Tags,
+		ListIDs:    camp.ListIDs,
+		Body:       camp.Body,
+		MediaIDs:   camp.MediaIDs,
+	}
+}
+
 // CampaignInput is what sync.go builds from a campaign's frontmatter+body to
 // create or update a listmonk campaign.
 type CampaignInput struct {
@@ -318,7 +336,23 @@ func (in CampaignInput) payload() map[string]any {
 		"lists":        in.ListIDs,
 		"content_type": "markdown",
 		"body":         in.Body,
+		// listmonk never derives a plaintext alternative from body/HTML on its
+		// own (models/campaigns.go only compiles AltBodyTpl, and
+		// internal/manager/message.go only emits a text/plain part, when
+		// altbody is explicitly non-null) — confirmed against the deployed
+		// v6.2.0 source. Reusing the same raw Markdown+template source as the
+		// altbody gets it personalized identically to body (AltBodyTpl compiles
+		// whenever the text contains {{ }} expressions), at the cost of
+		// Markdown syntax like *emphasis* showing up literally in plaintext
+		// clients — an acceptable, well-established tradeoff for Markdown.
+		"altbody": in.Body,
 		"type":    in.Type,
+		// listmonk's create/update handlers default an omitted messenger to
+		// "email" before validating, but the test-send handler validates the
+		// raw request body as-is — an omitted messenger there fails with
+		// "Unknown messenger .". eec-campaigns only ever sends email, so set
+		// it explicitly everywhere rather than relying on that asymmetry.
+		"messenger": "email",
 	}
 	if in.FromEmail != "" {
 		p["from_email"] = in.FromEmail
@@ -348,6 +382,7 @@ func parseCampaign(data []byte) (*Campaign, error) {
 		Status     string   `json:"status"`
 		Subject    string   `json:"subject"`
 		Body       string   `json:"body"`
+		AltBody    string   `json:"altbody"`
 		FromEmail  string   `json:"from_email"`
 		TemplateID int      `json:"template_id"`
 		Type       string   `json:"type"`
@@ -376,6 +411,7 @@ func parseCampaign(data []byte) (*Campaign, error) {
 		Status:     parsed.Status,
 		Subject:    parsed.Subject,
 		Body:       parsed.Body,
+		AltBody:    parsed.AltBody,
 		FromEmail:  parsed.FromEmail,
 		TemplateID: parsed.TemplateID,
 		Type:       parsed.Type,
@@ -482,14 +518,18 @@ func (c *Client) SetCampaignStatus(id int, status string) error {
 }
 
 // TestCampaign sends a preview of the campaign's current content to the
-// given addresses without touching its status. Request field confirmed
+// given addresses without touching its status. listmonk's test-send handler
-// against knadh/listmonk's actual handler source (cmd/campaigns.go's
+// binds the request into the same campReq struct create/update use and
-// campReq.SubscriberEmails, json tag "subscribers").
+// validates it in full, so the campaign's other fields (in particular a
-func (c *Client) TestCampaign(id int, emails []string) error {
+// non-empty "name") must be sent alongside "subscribers" — a request with
+// just {"subscribers": [...]} fails listmonk's own validation with "Invalid
+// length for name" before the "subscribers" field is ever looked at.
+func (c *Client) TestCampaign(id int, in CampaignInput, emails []string) error {
 	if len(emails) == 0 {
 		return nil
 	}
-	payload := map[string]any{"subscribers": emails}
+	payload := in.payload()
+	payload["subscribers"] = emails
 	respBody, status, err := c.do(http.MethodPost, fmt.Sprintf("/api/campaigns/%d/test", id), payload)
 	if err != nil {
 		return fmt.Errorf("sending test for campaign %d: %w", id, err)

internal/listmonk/listmonk_test.go

@@ -311,13 +311,17 @@ func TestTestCampaign_SendsSubscribersList(t *testing.T) {
 	defer srv.Close()
 
 	c := New(srv.URL, "u", "t")
-	if err := c.TestCampaign(7, []string{"me@example.com"}); err != nil {
+	in := CampaignInput{Name: "launch", Subject: "Hi", Body: "body", Type: "regular"}
+	if err := c.TestCampaign(7, in, []string{"me@example.com"}); err != nil {
 		t.Fatalf("TestCampaign: %v", err)
 	}
 	subs, ok := gotBody["subscribers"].([]any)
 	if !ok || len(subs) != 1 || subs[0] != "me@example.com" {
 		t.Errorf("expected subscribers=[me@example.com], got %+v", gotBody)
 	}
+	if gotBody["name"] != "launch" {
+		t.Errorf("expected the full campaign body (name=launch) alongside subscribers, got %+v", gotBody)
+	}
 }
 
 func TestErrorResponsesAreWrappedWithStatusAndBody(t *testing.T) {

main.go

@@ -81,6 +81,9 @@ func runSync(lm *listmonk.Client, path string) {
 	}
 	log.Printf("synced: %v", result.Synced)
 	log.Printf("unchanged: %v", result.Unchanged)
+	if len(result.Skipped) > 0 {
+		log.Printf("skipped (already sent): %v", result.Skipped)
+	}
 	if len(result.Rejected) > 0 {
 		log.Printf("rejected: %v", result.Rejected)
 	}