wagoe
Docs Why Wagoe Blog Home
Get Started

File storage abstraction with local filesystem, AWS S3 and Google Cloud Storage backends. Includes file validation, image processing, and signed URL generation.

Key namespaces

Namespace Purpose

wagoe.storage.core.validation

Pure: file validation (size, type, extension), filename sanitization

wagoe.storage.ports

Protocols: IFileStorage, IStreamingFileStorage, IUploadIndex, IExportJobStore, IImageProcessor

wagoe.storage.schema

Malli schemas: FileData, FileMetadata, StorageResult

wagoe.storage.shell.service

Service layer orchestrating validation + storage

wagoe.storage.shell.adapters.local

Local filesystem adapter (content-addressed with SHA-256)

wagoe.storage.shell.adapters.s3

AWS S3 / S3-compatible adapter (MinIO, DigitalOcean Spaces)

wagoe.storage.shell.adapters.gcs

Google Cloud Storage adapter

wagoe.storage.shell.adapters.image-processor

Java AWT image processing (no external deps)

wagoe.storage.shell.http-handlers

Ring handlers for upload/download/delete

Storage operations

(require '[wagoe.storage.ports :as ports])

(ports/store-file storage
  {:bytes (.getBytes "content") :content-type "text/plain"}
  {:filename "test.txt"})
;=> {:key "ab/1234-uuid-hash.txt" :url "..." :size 7 :content-type "text/plain"}

(ports/retrieve-file storage "ab/1234-uuid-hash.txt")
(ports/file-exists? storage "ab/1234-uuid-hash.txt")
(ports/delete-file storage "ab/1234-uuid-hash.txt")
(ports/generate-signed-url storage key 3600)  ; local signs only with :signing-secret

File validation

(require '[wagoe.storage.core.validation :as validation])

(validation/validate-file
  {:bytes file-bytes :content-type "image/png"}
  {:filename "photo.png"}
  {:max-size           (* 5 1024 1024)
   :allowed-types      #{"image/jpeg" "image/png" "image/webp"}
   :allowed-extensions #{"jpg" "jpeg" "png" "webp"}})

Image processing

(ports/resize-image processor file-bytes {:width 800 :height 600}) ; nil keeps the aspect ratio
(ports/create-thumbnail processor file-bytes 150)                   ; longest side, in pixels

Storage key format

  • Local: {shard}/{timestamp}-{uuid}-{hash16}.{ext} (shard = first 2 chars of SHA-256)

  • S3: {prefix}/{timestamp}-{uuid8}.{ext}

Configuration

:wagoe/storage in :active, chosen by :provider (default :local).

;; Local (development) — :root or :base-path
:wagoe/storage {:provider :local :root "/var/uploads"}

;; S3 (production); :endpoint for S3-compatible services
:wagoe/storage {:provider :s3 :bucket #env AWS_S3_BUCKET :region #env AWS_REGION
                :access-key #env AWS_ACCESS_KEY_ID :secret-key #env AWS_SECRET_ACCESS_KEY}

;; Google Cloud Storage
:wagoe/storage {:provider :gcs :bucket "b" :project-id "p"}

Upload index

Every upload through the storage service is recorded in the storage_uploads table (IUploadIndex): key, owner user and tenant, size, content type and when it was stored. The upload routes take the owner from the request; other callers pass :owner {:user-id … :tenant-id …} in the metadata. An upload the index cannot record fails, and its file is deleted. Files stored before the index existed, or straight through IFileStorage, are not in it.

The index feeds two privacy contributions:

  • a :storage section in a user’s or tenant’s data export, listing their uploads; a zip export includes the files.

  • the :storage/uploads retention policy, which deletes uploads stored before its period, file and record:

:wagoe/privacy {:retention {:storage/uploads {:retain "P1Y"}}}

Data export as a background job

With the jobs module enabled, a signed-in user can have their export built in the background rather than during the request, for a subject whose zip takes too long to stream:

  • POST /api/v1/me/export-jobs queues it (202). While one is open, asking again returns the same job.

  • GET /api/v1/me/export-jobs/:id reports pending, running, done or failed, and when done a download-path.

  • GET /api/v1/me/export-jobs/:id/download serves the zip for 24 hours. On S3 and GCS it redirects to a signed URL that lives five minutes; on local storage the route sends the file.

Platform admins ask for someone else’s export the same way, at POST /api/v1/users/:id/export-jobs and POST /api/v1/tenants/:id/export-jobs, and follow it at the Location they get back, under /api/v1/me/export-jobs. The export is written to the audit log (the user’s, or tenant_audit_log) when it is asked for, because that is where the admin and their request are known. The admin can follow the job and download it only while they are a platform admin.

Only the user who asked sees a job; to anyone else it does not exist. A user has at most one open job per subject, even when requests race. A user may open five an hour (:export-requests-per-hour in :wagoe/storage, 0 for no limit); more answer 429. The archive is streamed to a temporary file under the storage root’s .spool/ and stored private under data-exports/; it is not an upload. A spool left by a crash is deleted at the next boot. Jobs run on the :default queue, or :export-queue in :wagoe/storage.

Nothing deletes old archives until you configure the :storage/export-archives retention policy. There is no default because retention in Wagoe is always opt-in: an upgrade must not start deleting data. After 24 hours an archive cannot be downloaded, but it still holds personal data, so configure the policy when you enable export jobs:

:wagoe/privacy {:retention {:storage/export-archives {:retain "P2D"}}}

Testing

clojure -M:test :storage