Fix campaign/listmonk API shapes against confirmed source, not guesses

Read knadh/listmonk's actual Go source (cmd/campaigns.go, cmd/subscribers.go)
instead of relying on incomplete public docs. Two corrections: the media
attach field is "media" (plain IDs), not "media_ids" as originally guessed;
and there's a single query-based bulk list action
(PUT /api/subscribers/query/lists) that applies a segment_query server-side,
so resolveSegment no longer fetches subscriber IDs client-side before
bulk-adding them. Create-status-defaults-to-draft and the test-send
"subscribers" field are now confirmed rather than flagged TODO(verify).
This commit is contained in:
2026-07-10 08:37:07 -04:00
parent b16d2b6c5f
commit bcf8df5579
5 changed files with 109 additions and 166 deletions
+9 -9
View File
@@ -50,12 +50,12 @@ maintain.
## Segmentation
`segment_query` is a raw Postgres-style SQL boolean expression — the same segmentation mechanism listmonk's own admin UI search box already accepts (`AND`/`OR`/`NOT`, `IN`, `LIKE`, `EXISTS` subqueries against e.g. `campaign_views` for engagement-based segments). No custom query language here; the string is passed straight through to listmonk's own subscriber-query API.
`segment_query` is a raw Postgres-style SQL boolean expression — the same segmentation mechanism listmonk's own admin UI search box already accepts (`AND`/`OR`/`NOT`, `IN`, `LIKE`, `EXISTS` subqueries against e.g. `campaign_views` for engagement-based segments). No custom query language here; the string is passed straight through to listmonk's query-based bulk list action.
Since a listmonk campaign can only target whole list(s), not an arbitrary query directly, `sync` materializes the query into a managed list on your behalf:
1. Query subscribers matching the expression.
2. Bulk-add matches to an auto-created/reused list named `segment:<slug>`.
1. Find or create a list named `segment:<slug>`.
2. Ask listmonk to add every subscriber matching the expression to that list, in one call — no subscriber IDs are ever fetched client-side, listmonk applies the query and the list membership change server-side.
3. Target the campaign at that list, alongside any named `lists:` also given.
That list is a point-in-time snapshot taken at sync time, not a live dynamic segment — push again before sending to pick up newly-matching subscribers. An invalid SQL fragment fails that campaign's sync with listmonk's own error message; nothing is created or sent.
@@ -92,14 +92,14 @@ One bad campaign in the push doesn't block the others — check `rejected:` in t
listmonk's only currently-provisioned email template (`eec-passthrough`) is a bare passthrough used for `eec`'s transactional course emails — no unsubscribe footer, no branding. Create (or confirm) a real campaign template with a proper unsubscribe link in the listmonk admin UI before running your first `send`; drafts and previews render fine without one, but a real send without it isn't compliant.
## Open items / verify against the live instance
## Open items
A few request shapes in `internal/listmonk` are marked `// TODO(verify)` because they aren't confirmed against public listmonk API docs — check them against the actual running instance/version (e.g. by inspecting the admin UI's own network requests) before depending on them in a real send:
Every request shape in `internal/listmonk` is now confirmed against `knadh/listmonk`'s actual Go source (not just its docs, which are incomplete on a few of these): the campaign create/update payload, its `media` field for attachments (listmonk's request/response asymmetry — requests send plain IDs under `media`, responses echo full objects back under the same key), create always defaulting to `draft` regardless of any caller-supplied status, the test-send endpoint's `subscribers` field, and the query-based bulk list action (`PUT /api/subscribers/query/lists`) segmentation uses. What's left is genuinely operational, not code:
- The field for attaching uploaded media IDs to a campaign (currently assumed `media_ids`).
- The bulk subscriber→list action's exact endpoint/body shape (currently `PUT /api/subscribers/lists`).
- The test-send endpoint's recipient field (currently assumed `subscribers`).
- Whether `POST /api/campaigns` needs/accepts an explicit `status` field, and that its create-time default really is `draft`.
- **A real, unsubscribe-capable campaign template** needs to exist in the live listmonk instance before a real send (see above) — a content/admin-UI task, not something this tool verifies for you.
- **Provisioning the dedicated listmonk API user and pasting its token into this repo's Gitea secrets** (see Deploying below) hasn't happened yet.
- **Gitea Actions' tag-push and `workflow_dispatch` triggers** are standard, long-supported Actions syntax and the runner already successfully uses `uses: actions/checkout@v4` elsewhere in this org, so this should work as written — but it's still worth confirming the first time `send.yml` actually fires.
- The source was read against `knadh/listmonk`'s `master` branch; if the live instance runs a substantially older or newer version, a quick diff against its own `cmd/campaigns.go`/`cmd/subscribers.go` is cheap insurance before the first real send.
## Deploying
+7 -9
View File
@@ -157,21 +157,19 @@ func resolveLists(lm *listmonk.Client, names []string) ([]int, error) {
// resolveSegment materializes a segment_query into list membership, since a
// listmonk campaign can only target whole list(s), not an arbitrary query
// directly (still an open, unmerged upstream feature request — see the
// plan's Open items). The list is a point-in-time snapshot: re-running sync
// re-evaluates the query and re-syncs membership, so a later push before
// send picks up newly-matching subscribers.
// plan's Open items). listmonk applies the query server-side in one call
// (QueryAddToList) — no subscriber IDs are ever fetched client-side. The
// list is a point-in-time snapshot: re-running sync re-evaluates the query
// and re-syncs membership, so a later push before send picks up
// newly-matching subscribers.
func resolveSegment(lm *listmonk.Client, slug, query string) (int, error) {
ids, err := lm.QuerySubscriberIDs(query)
if err != nil {
return 0, fmt.Errorf("segment_query: %w", err)
}
listName := "segment:" + slug
listID, err := lm.FindOrCreateListByName(listName)
if err != nil {
return 0, fmt.Errorf("segment list %q: %w", listName, err)
}
if err := lm.BulkAddToList(ids, listID); err != nil {
return 0, fmt.Errorf("segment list %q: %w", listName, err)
if err := lm.QueryAddToList(query, listID); err != nil {
return 0, fmt.Errorf("segment_query: %w", err)
}
return listID, nil
}
+35 -38
View File
@@ -49,8 +49,8 @@ type testCall struct {
Emails []string
}
type bulkAddCall struct {
IDs []int
type segmentQueryCall struct {
Query string
ListID int
}
@@ -61,13 +61,13 @@ type fakeListmonk struct {
lists []fakeList
campaigns []fakeCampaign
media []fakeMedia
subscriberIDsForQuery map[string][]int
testCalls []testCall
bulkAddCalls []bulkAddCall
segmentQueryCalls []segmentQueryCall
failSegmentQuery bool
}
func newFakeListmonk(t *testing.T) *fakeListmonk {
return &fakeListmonk{t: t, nextID: 1, subscriberIDsForQuery: map[string][]int{}}
return &fakeListmonk{t: t, nextID: 1}
}
func (f *fakeListmonk) id() int {
@@ -105,10 +105,8 @@ func (f *fakeListmonk) handle(w http.ResponseWriter, r *http.Request) {
f.writeMedia(w)
case r.Method == http.MethodPost && r.URL.Path == "/api/media":
f.uploadMedia(w, r)
case r.Method == http.MethodGet && r.URL.Path == "/api/subscribers":
f.querySubscribers(w, r)
case r.Method == http.MethodPut && r.URL.Path == "/api/subscribers/lists":
f.bulkAdd(w, r)
case r.Method == http.MethodPut && r.URL.Path == "/api/subscribers/query/lists":
f.queryAddToList(w, r)
default:
f.t.Errorf("fakeListmonk: unhandled request %s %s", r.Method, r.URL.Path)
w.WriteHeader(http.StatusNotFound)
@@ -120,10 +118,14 @@ func (c fakeCampaign) toJSON() map[string]any {
for _, id := range c.ListIDs {
lists = append(lists, map[string]any{"id": id})
}
media := make([]map[string]any, 0, len(c.MediaIDs))
for _, id := range c.MediaIDs {
media = append(media, map[string]any{"id": id})
}
return map[string]any{
"id": c.ID, "name": c.Name, "status": c.Status, "subject": c.Subject,
"body": c.Body, "from_email": c.FromEmail, "template_id": c.TemplateID,
"type": c.Type, "tags": c.Tags, "lists": lists, "media_ids": c.MediaIDs,
"type": c.Type, "tags": c.Tags, "lists": lists, "media": media,
}
}
@@ -142,7 +144,7 @@ func (f *fakeListmonk) createCampaign(w http.ResponseWriter, r *http.Request) {
Subject: str(body["subject"]), Body: str(body["body"]),
FromEmail: str(body["from_email"]), TemplateID: toInt(body["template_id"]),
Type: str(body["type"]), Tags: toStrings(body["tags"]),
ListIDs: toInts(body["lists"]), MediaIDs: toInts(body["media_ids"]),
ListIDs: toInts(body["lists"]), MediaIDs: toInts(body["media"]),
}
f.campaigns = append(f.campaigns, c)
json.NewEncoder(w).Encode(map[string]any{"data": c.toJSON()})
@@ -160,7 +162,7 @@ func (f *fakeListmonk) updateCampaign(w http.ResponseWriter, r *http.Request) {
f.campaigns[i].Type = str(body["type"])
f.campaigns[i].Tags = toStrings(body["tags"])
f.campaigns[i].ListIDs = toInts(body["lists"])
f.campaigns[i].MediaIDs = toInts(body["media_ids"])
f.campaigns[i].MediaIDs = toInts(body["media"])
json.NewEncoder(w).Encode(map[string]any{"data": f.campaigns[i].toJSON()})
return
}
@@ -226,24 +228,19 @@ func (f *fakeListmonk) uploadMedia(w http.ResponseWriter, r *http.Request) {
json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"id": m.ID}})
}
func (f *fakeListmonk) querySubscribers(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query().Get("query")
ids := f.subscriberIDsForQuery[q]
results := make([]map[string]any, 0, len(ids))
for _, id := range ids {
results = append(results, map[string]any{"id": id})
func (f *fakeListmonk) queryAddToList(w http.ResponseWriter, r *http.Request) {
if f.failSegmentQuery {
w.WriteHeader(http.StatusBadRequest)
w.Write([]byte(`{"message":"invalid SQL expression"}`))
return
}
json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"results": results}})
}
func (f *fakeListmonk) bulkAdd(w http.ResponseWriter, r *http.Request) {
body := decodeBody(r)
targetIDs := toInts(body["target_list_ids"])
var listID int
if len(targetIDs) > 0 {
listID = targetIDs[0]
}
f.bulkAddCalls = append(f.bulkAddCalls, bulkAddCall{IDs: toInts(body["ids"]), ListID: listID})
f.segmentQueryCalls = append(f.segmentQueryCalls, segmentQueryCall{Query: str(body["query"]), ListID: listID})
w.Write([]byte(`{}`))
}
@@ -434,7 +431,6 @@ func TestSyncDir_SegmentQueryMaterializesIntoManagedList(t *testing.T) {
f := newFakeListmonk(t)
f.lists = []fakeList{{ID: 3, Name: "Newsletter"}}
query := "subscribers.attribs->>'source' = 'workshop'"
f.subscriberIDsForQuery[query] = []int{10, 11, 12}
lm := f.client()
root := t.TempDir()
@@ -448,11 +444,11 @@ func TestSyncDir_SegmentQueryMaterializesIntoManagedList(t *testing.T) {
if len(result.Rejected) != 0 {
t.Fatalf("expected no rejections, got %v", result.Rejected)
}
if len(f.bulkAddCalls) != 1 {
t.Fatalf("expected 1 bulk-add call, got %d", len(f.bulkAddCalls))
if len(f.segmentQueryCalls) != 1 {
t.Fatalf("expected 1 segment query call, got %d", len(f.segmentQueryCalls))
}
if len(f.bulkAddCalls[0].IDs) != 3 {
t.Errorf("expected 3 subscriber IDs bulk-added, got %v", f.bulkAddCalls[0].IDs)
if f.segmentQueryCalls[0].Query != query {
t.Errorf("expected the query sent verbatim, got %q", f.segmentQueryCalls[0].Query)
}
var segList *fakeList
@@ -464,6 +460,9 @@ func TestSyncDir_SegmentQueryMaterializesIntoManagedList(t *testing.T) {
if segList == nil {
t.Fatal("expected a managed 'segment:launch' list to be created")
}
if f.segmentQueryCalls[0].ListID != segList.ID {
t.Errorf("expected the segment query call to target list %d, got %d", segList.ID, f.segmentQueryCalls[0].ListID)
}
if len(f.campaigns) != 1 {
t.Fatalf("expected 1 campaign, got %d", len(f.campaigns))
}
@@ -478,27 +477,25 @@ func TestSyncDir_SegmentQueryMaterializesIntoManagedList(t *testing.T) {
}
}
func TestSyncDir_InvalidSegmentQueryRejectsWithoutSideEffects(t *testing.T) {
func TestSyncDir_InvalidSegmentQueryRejectsWithoutCreatingCampaign(t *testing.T) {
f := newFakeListmonk(t)
f.lists = []fakeList{{ID: 3, Name: "Newsletter"}}
f.failSegmentQuery = true
lm := f.client()
root := t.TempDir()
// No entry seeded in subscriberIDsForQuery for this exact string simulates
// a query listmonk would reject — here we just confirm an empty/no-match
// result still flows through cleanly without creating a segment list
// mistakenly treated as an error path; a real invalid-SQL rejection from
// listmonk itself surfaces as a plain HTTP-error wrapped by
// QuerySubscriberIDs, exercised at the listmonk package's own test level.
fm := baseFrontmatter + "segment_query: \"subscribers.attribs->>'nope' = 'nothing'\"\n"
fm := baseFrontmatter + "segment_query: \"not valid sql\"\n"
writeCampaignDir(t, root, "launch", fm, "Hello world.")
result, err := SyncDir(lm, root, nil)
if err != nil {
t.Fatalf("SyncDir: %v", err)
}
if len(result.Rejected) != 0 {
t.Fatalf("expected no rejections for a zero-match segment query, got %v", result.Rejected)
if len(result.Rejected) != 1 || !strings.Contains(result.Rejected[0], "segment_query") {
t.Fatalf("expected a segment_query rejection, got %v", result.Rejected)
}
if len(f.campaigns) != 0 {
t.Errorf("expected no campaign created when segment_query fails, got %d", len(f.campaigns))
}
}
+36 -68
View File
@@ -16,7 +16,6 @@ import (
"io"
"mime/multipart"
"net/http"
"net/url"
)
type Client struct {
@@ -160,60 +159,27 @@ func (c *Client) FindOrCreateListByName(name string) (int, error) {
// ---- Subscribers / segmentation ----
// QuerySubscriberIDs runs a raw SQL boolean expression against the
// subscribers table (the same segmentation mechanism listmonk's own admin
// UI search box uses) and returns the IDs of every match. per_page=all
// deliberately skips pagination, matching drip's existing Query method.
func (c *Client) QuerySubscriberIDs(query string) ([]int, error) {
path := "/api/subscribers?per_page=all&query=" + url.QueryEscape(query)
respBody, status, err := c.do(http.MethodGet, path, nil)
if err != nil {
return nil, fmt.Errorf("querying subscribers: %w", err)
}
if status != http.StatusOK {
return nil, fmt.Errorf("segment_query failed (%d): %s", status, string(respBody))
}
var parsed struct {
Data struct {
Results []struct {
ID int `json:"id"`
} `json:"results"`
} `json:"data"`
}
if err := json.Unmarshal(respBody, &parsed); err != nil {
return nil, fmt.Errorf("parsing subscriber query response: %w", err)
}
ids := make([]int, 0, len(parsed.Data.Results))
for _, r := range parsed.Data.Results {
ids = append(ids, r.ID)
}
return ids, nil
}
// BulkAddToList adds the given subscriber IDs to listID.
//
// TODO(verify): the bulk list-membership action's exact endpoint/body shape
// isn't confirmed against a live listmonk instance — this targets the
// documented bulk subscriber-action endpoint (PUT /api/subscribers/lists
// with explicit ids+action+target_list_ids), which is the same mechanism
// listmonk's admin UI uses for "add selected subscribers to list". Confirm
// against the real instance/version before relying on this in production.
func (c *Client) BulkAddToList(ids []int, listID int) error {
if len(ids) == 0 {
return nil
}
// QueryAddToList bulk-adds every subscriber matching the given raw SQL
// boolean expression to listID, in one call — listmonk's query-based bulk
// subscriber-list action, confirmed against the actual handler source
// (cmd/subscribers.go's ManageSubscriberListsByQuery, routed at
// PUT /api/subscribers/query/lists, request fields query/target_list_ids/
// action/status). This is the same segmentation mechanism listmonk's own
// admin UI search box uses, applied server-side without ever fetching
// individual subscriber IDs client-side.
func (c *Client) QueryAddToList(query string, listID int) error {
payload := map[string]any{
"ids": ids,
"action": "add",
"query": query,
"target_list_ids": []int{listID},
"action": "add",
"status": "unconfirmed",
}
respBody, status, err := c.do(http.MethodPut, "/api/subscribers/lists", payload)
respBody, status, err := c.do(http.MethodPut, "/api/subscribers/query/lists", payload)
if err != nil {
return fmt.Errorf("bulk-adding %d subscriber(s) to list %d: %w", len(ids), listID, err)
return fmt.Errorf("segment_query bulk-add to list %d: %w", listID, err)
}
if status != http.StatusOK {
return fmt.Errorf("bulk-adding subscribers to list %d failed (%d): %s", listID, status, string(respBody))
return fmt.Errorf("segment_query bulk-add to list %d failed (%d): %s", listID, status, string(respBody))
}
return nil
}
@@ -364,12 +330,13 @@ func (in CampaignInput) payload() map[string]any {
p["tags"] = in.Tags
}
if len(in.MediaIDs) > 0 {
// TODO(verify): field name for attaching media library items to a
// campaign isn't documented publicly — confirm against a live
// listmonk admin UI network trace (attach a file to any test
// campaign and inspect the PUT /api/campaigns/{id} request it
// sends) before relying on this in production.
p["media_ids"] = in.MediaIDs
// Field name confirmed against knadh/listmonk's actual request-binding
// struct (cmd/campaigns.go's campReq: `MediaIDs []int json:"media"`) —
// requests take plain media IDs under "media"; listmonk's own
// responses echo full media objects back under the same key (see
// parseCampaign below), a request/response asymmetry on listmonk's
// side, not a mistake here.
p["media"] = in.MediaIDs
}
return p
}
@@ -388,7 +355,9 @@ func parseCampaign(data []byte) (*Campaign, error) {
Lists []struct {
ID int `json:"id"`
} `json:"lists"`
MediaIDs []int `json:"media_ids"`
Media []struct {
ID int `json:"id"`
} `json:"media"`
}
if err := json.Unmarshal(data, &parsed); err != nil {
return nil, err
@@ -397,6 +366,10 @@ func parseCampaign(data []byte) (*Campaign, error) {
for _, l := range parsed.Lists {
listIDs = append(listIDs, l.ID)
}
mediaIDs := make([]int, 0, len(parsed.Media))
for _, m := range parsed.Media {
mediaIDs = append(mediaIDs, m.ID)
}
return &Campaign{
ID: parsed.ID,
Name: parsed.Name,
@@ -408,7 +381,7 @@ func parseCampaign(data []byte) (*Campaign, error) {
Type: parsed.Type,
Tags: parsed.Tags,
ListIDs: listIDs,
MediaIDs: parsed.MediaIDs,
MediaIDs: mediaIDs,
}, nil
}
@@ -452,13 +425,10 @@ func (c *Client) FindCampaignByName(name string) (*Campaign, error) {
}
}
// CreateCampaign creates a new campaign. listmonk's create-time default
// status is draft, which is this tool's core safety invariant — see the
// TODO below.
//
// TODO(verify): confirm that POST /api/campaigns' create-time default
// status really is "draft" against a live instance (and whether an explicit
// status field is even accepted on create) before depending on it.
// CreateCampaign creates a new campaign. Confirmed against knadh/listmonk's
// actual handler source: the create handler ignores any caller-supplied
// status and always creates as draft — this tool's core safety invariant —
// so in.payload() doesn't bother sending one.
func (c *Client) CreateCampaign(in CampaignInput) (*Campaign, error) {
respBody, status, err := c.do(http.MethodPost, "/api/campaigns", in.payload())
if err != nil {
@@ -512,11 +482,9 @@ func (c *Client) SetCampaignStatus(id int, status string) error {
}
// TestCampaign sends a preview of the campaign's current content to the
// given addresses without touching its status.
//
// TODO(verify): the request body key for the recipient address list isn't
// confirmed against a live instance — using "subscribers" per listmonk's
// documented shape for this endpoint; confirm before relying on it.
// given addresses without touching its status. Request field confirmed
// against knadh/listmonk's actual handler source (cmd/campaigns.go's
// campReq.SubscriberEmails, json tag "subscribers").
func (c *Client) TestCampaign(id int, emails []string) error {
if len(emails) == 0 {
return nil
+15 -35
View File
@@ -98,28 +98,7 @@ func TestFindOrCreateListByName_CreatesWhenMissing(t *testing.T) {
}
}
func TestQuerySubscriberIDs_UsesPerPageAllAndParsesIDs(t *testing.T) {
var gotPath string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotPath = r.URL.RequestURI()
w.Write([]byte(`{"data":{"results":[{"id":1},{"id":2}]}}`))
}))
defer srv.Close()
c := New(srv.URL, "u", "t")
ids, err := c.QuerySubscriberIDs("subscribers.attribs->>'source' = 'workshop'")
if err != nil {
t.Fatalf("QuerySubscriberIDs: %v", err)
}
if !strings.Contains(gotPath, "per_page=all") {
t.Errorf("expected per_page=all, got %q", gotPath)
}
if len(ids) != 2 || ids[0] != 1 || ids[1] != 2 {
t.Errorf("expected [1 2], got %v", ids)
}
}
func TestBulkAddToList_SendsExpectedPayload(t *testing.T) {
func TestQueryAddToList_SendsExpectedPayload(t *testing.T) {
var gotMethod, gotPath string
var gotBody map[string]any
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -132,34 +111,35 @@ func TestBulkAddToList_SendsExpectedPayload(t *testing.T) {
defer srv.Close()
c := New(srv.URL, "u", "t")
if err := c.BulkAddToList([]int{1, 2, 3}, 9); err != nil {
t.Fatalf("BulkAddToList: %v", err)
query := "subscribers.attribs->>'source' = 'workshop'"
if err := c.QueryAddToList(query, 9); err != nil {
t.Fatalf("QueryAddToList: %v", err)
}
if gotMethod != http.MethodPut {
t.Errorf("expected PUT, got %s", gotMethod)
}
if gotPath != "/api/subscribers/lists" {
t.Errorf("expected /api/subscribers/lists, got %q", gotPath)
if gotPath != "/api/subscribers/query/lists" {
t.Errorf("expected /api/subscribers/query/lists, got %q", gotPath)
}
if gotBody["query"] != query {
t.Errorf("expected query sent verbatim, got %+v", gotBody)
}
if gotBody["action"] != "add" {
t.Errorf("expected action=add, got %+v", gotBody)
}
}
func TestBulkAddToList_NoopOnEmptyIDs(t *testing.T) {
var calls int
func TestQueryAddToList_SurfacesListmonkErrors(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
w.Write([]byte(`{}`))
w.WriteHeader(http.StatusBadRequest)
w.Write([]byte(`{"message":"invalid SQL expression"}`))
}))
defer srv.Close()
c := New(srv.URL, "u", "t")
if err := c.BulkAddToList(nil, 9); err != nil {
t.Fatalf("BulkAddToList: %v", err)
}
if calls != 0 {
t.Errorf("expected no request for empty id list, got %d", calls)
err := c.QueryAddToList("not valid sql", 9)
if err == nil || !strings.Contains(err.Error(), "invalid SQL expression") {
t.Fatalf("expected listmonk's error message to surface, got %v", err)
}
}