Skip to main content

#3692 Polycons Framework (approved)

⚠️ Note: polycons was an early design for how to support CDK constructs that work across multiple cloud targets. It has since been archived to [https://github.com/winglang/polycons], but we've kept the RFC here for historical value. Check out the SDK Architecture to see where many ideas from this RFC were adopted!

  • Author(s):: @MarkMcCulloh
  • Submission Date: 2022-06-14
  • Stage: Approved
  • Stage Date: 2022-07-20
  • Implementation: Repository

This document describes polycons, a dependency injection meta-framework for polymorphic constructs. After some initial sections, this is structured as a proposed README to demonstrate the expected usage and general concepts.

Goal​

Enable a JSII-compatible language to write portable abstract construct-based applications, including "runtime processes" that can interact with these abstractions.

Scope​

These things fall within scope of the framework. They may not necessarily be in the same package.

  • Registration of polycon objects
  • Resolution system for polycon concretization
  • "Process" abstraction for runtime code description
  • System to inject information from construction-time into running processes

Omissions​

  • The concept of "Cloud Debug Symbols" is omitted from this RFC, however it very reasonably falls within the interest of this framework. The discussion of that deserves its own RFC.
  • There are intentions for a more advanced polycon resolution system, but the current form in this document has been simplified. Iterations of that system should have its own RFC.

Proposed README below

Polycons

A meta CDK framework for building portable constructs.

Why?​

When working in existing CDKs (aws-cdk, cdktf, cdk8s, etc.), constructs immediately become tied to their provisioning system/format (cloudformation, terraform, etc) and their eventual target platform (AWS, Kubernetes, etc).

Polycons represent a resource whose implementation is injected during construction to create truly portable constructs.

Such as:

  • Function - AWS Lambda or a GCP Cloud Function
  • Bucket - AWS S3 Bucket or Cloudflare R2 storage
  • Queue - In-memory queue running in Node or SQS Queue

This polymorphism extends to interactions in your runtime code, allowing for the creation of implementation-agnostic and logic-focused applications.

Note: This framework is not a CDK, it operates on a layer above constructs and intends to utilize existing CDKs and provisioning engines to do what they do best.

Usage​

Polymorphic Constructs​

// construct.ts

import { Queue } from "pocix";
import { Construct } from "constructs";

export class SeededQueue extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const queue = new Queue(this, "Queue");

    // Seed the Queue during creation
    queue.enqueue({ message: "Hello Polycons!" });
  }
}

This construct contains a Queue resource from a library of polycons. This queue could be any implementation deployed to any cloud (or maybe not even a cloud at all!). This construct is now completely portable!

Runtime Clients​

Very often you will want to interact with a polycon construct in your runtime code. In the framework, this interop is called a "capture" and the result is similarly polymorphic. This means the code below would work just as well deployed to a Lambda in AWS to push to SQS, as it would running in a local Node process pushing to a global in-memory queue.

// app.ts

export class QueuePusher extends Construct {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const myQueue = new Queue(this, "Queue");
    
    const func = new Function(this, "Function", {
      process: {
        code: Code.fromFile(join(__dirname, "queue-code.ts")),
        entrypoint: "handler",
        captures: [
          Capture.client({
            // The name used to reference this capture in the runtime code
            name: "queue",
            // The polycon to bind to the runtime
            target: myQueue,
            // The methods intended to be used
            methods: ["push"],
          }),
        ],
      }
    });
  }
}
// queue-code.ts

import { QueueClient } from "pocix-clients";

interface Captures {
  queue: QueueClient;
}

// This handler will be invoked by a shim containing clients for the captures
export async function handler(_event: any, captures: Captures) {
  // The client implementation knows how to talk to the queue you care about!
  await captures.queue.push({ message: "Hello Polycons!" });
}

Creating Polycons​

A polycon is defined by:

  • a qualifier string constant, globally unique among all polycons
  • a JSII struct specifying required/optional properties for instantiating the construct
  • a JSII behavioral interface consisting of properties and methods for use during construction
  • (Optional) a JSII behavioral interface consisting of methods for use during runtime

Define interfaces​

There are three important interfaces to define when creating a polycon:

Construction API​

This behavioral interface defines the public properties and methods associated with a given construct. This interface extends IConstruct.

export interface IBucket extends IConstruct {
  readonly isPublic: boolean;
  addUploadHandler(func: IFunction): void
}

Properties​

The properties to define the construct during instantiation.

export interface BucketProps {
  readonly public?: boolean;
}

Client API​

To ensure a polycon is portable outside construction-time, we must define an API to interact with it at runtime. This provides predictable behavior and a consistent contract regardless of the construct or the client implementation. By convention, this API will be considered the default for any interactions.

export interface IBucketClient {
  download(request: DownloadRequest): Promise<DownloadResponse>;
  upload(request: UploadRequest): Promise<UploadResponse>;
}

Declare the proxy class​

We need a class that allows us to instantiate the underlying implementation. While this class is not abstract, note that the return value of the constructor will be a different class (the concrete implementation). Mutations to the class will not affect the implementation unless explicitly interacting with the return value super(...).

All polycons must uniquely identify themselves with a qualifier, whose value must be globally unique. By convention, this value is the "fully qualified type" of the polycon. For example, if the package name is pocix, it's in the cloud module/namespace, and the name is Bucket, then the qualifier would be pocix.cloud.Bucket. During instantiation, this uniqueness will be verified in the base Polycon to prevent collisions.

The example below is the full Bucket polycon declaration with the class included.

Note: This file is mostly boilerplate, except for the name and the interface properties/types. A [projen] component is available to make generating polycons simple.

// bucket.ts

// Unique global identifier for this polycon
export const BUCKET_QUALIFIER = "pocix.Bucket";

export interface IBucket extends IConstruct {
  readonly isPublic: boolean;
  addUploadHandler(func: IFunction): void;
}

// Construction properties for your polycon
export interface BucketProps {
  readonly public?: boolean;
}

export class Bucket extends Polycon implements IBucket {
  public get isPublic(): boolean {
    // This method is not the real implementation and should not be reachable
    throw this.proxyError("public");
  }

  public addUploadHandler(func: IFunction): void {
    throw this.proxyError("addUploadHandler");
  }

  constructor(scope: Construct, id: string, props?: BucketProps) {
    super(BUCKET_QUALIFIER, scope, id, props);
  }
}

Implementing polycons​

Once the polycon is fully declared, we can define a concrete implementation. This is just a Construct that implements the desired Polycon's interface and takes the same properties in the constructor. Many of these can exist and each can be composed of any type of construct (from any CDK!).

The concrete implementation will be chosen automatically through a PolyconFactory (see below) when instantiating the polycon. You may also instantiate the concrete implementation yourself instead to bypass the factory, just make sure to add a CaptureClient as well.

S3 Bucket (cdktf) example implementation of a Bucket polycon:

// aws-bucket.ts

import {
  S3Bucket,
  S3BucketPublicAccessBlock,
} from "@cdktf/provider-aws/lib/s3";
import { Construct } from "constructs";
import { BucketProps, IBucket } from "../../pocix";

export class AWSBucket extends Construct implements IBucket {
  public readonly isPublic: boolean;
  private readonly bucket: S3Bucket;

  constructor(scope: Construct, id: string, props?: BucketProps) {
    super(scope, id);

    this.isPublic = props?.public ?? false;

    // Create underlying cdktf bucket
    this.bucket = new S3Bucket(this, "Bucket");

    new S3BucketPublicAccessBlock(this, "BlockPublicAccess", {
      bucket: this.bucket.bucket,
      blockPublicAcls: !this.isPublic,
      blockPublicPolicy: !this.isPublic,
      ignorePublicAcls: !this.isPublic,
      restrictPublicBuckets: !this.isPublic,
    });
  }
}

Client implementation​

To define a client available during construction you must implement ICaptureClient to provide three important functionalities:

renderCapture(capture: Capture): string

Returns a string representation of the client code. This is used to generate the value of the Captures object during runtime. For capturing primitives, this could simply be the serialized capture value. For many JS clients this is an invocation of the imported module to set up a client object. With this, you can add additional arguments to pass to your client implementation based on the capture itself.

bindToComputePlatform(capture: Capture, consumer: IComputePlatform): void

To ensure a client can run successfully it may need to define additional construction-time configuration for the compute system. Some examples:

  • Add an environment variable with a unique external id to use later in the client
  • Ensure the consumer has the required permissions to access the capture (e.g., Mutate policies for IAM role of the consumer Lambda to access the desired S3 bucket)

get code(): Code | undefined

If the client has code it needs to include for any consumers, this exposes it to be available to any consuming Processes so they can be aware that it's a dependency.

// bucket-client.ts

// Ensure consistent environment variable name
function getEnvName(capture: Capture) {
  return `__CAPTURE_SYM_${capture.name}`;
}

export class AWSSDKBucketCaptureClient implements ICaptureClient {
  // This Code points to an actual implementation of a bucket client with download/upload/etc. methods
  public get code(): Code {
    return Code.fromFile(join(__dirname, "runtime/bucket-client.ts")),
  }

  public renderCapture(capture: Capture): string {
    // The client module should be available in the process to be consumed through the entrypoint
    return `${this.code.entrypoint}(process.env["${getEnvName(
      capture
    )}"])`;
  }

  public bindToComputePlatform(capture: Capture, consumer: IComputePlatform): void {
    const target = capture.target as Bucket;
    consumer.setEnvironment(this.getEnvName(capture), target.bucket.arn);

    // Any additional compute platforms may need custom logic not provided by the common interface
    if (consumer instanceof TFLambdaFunction) {
      consumer.lambdaRole.putInlinePolicy([
        {
          policy: JSON.stringify({
            Version: "2012-10-17",
            Statement: [
              {
                // Hopefully this isn't your final policy :)
                Action: "s3:*",
                Effect: "Allow",
                Resource: target.bucket.arn,
              },
            ],
          }),
        },
      ]);
    }
  }
}

Factories​

To tie everything together, a polycon factory is responsible for resolving every polycon found in a construct tree with its concrete implementation. For example, an AWS CDK factory maps Bucket to S3 Bucket and Queue to SQS Queue and so on. It also ensures constructs like Bucket have an applicable S3 AWS SDK client available on the construct.

// tf-aws-factory.ts
import * as std from "pocix";
import { Construct } from "constructs";
import { PolyconFactory, CaptureClient } from "polycons";
import { TFBucket } from "./resources/tf-bucket.ts";

export class TerraformAWSFactory extends PolyconFactory {
  public resolve(
    qualifier: string,
    scope: Construct,
    id: string,
    props?: any
  ): Construct {
    switch (qualifier) {
      case std.BUCKET_QUALIFIER:
        const bucket = new TFBucket(scope, id);
        CaptureClient.register(bucket, new AWSSDKBucketCaptureClient());
        return bucket;
      // ... other polycon cases ...
      default:
        throw new Error(`Qualifier ${qualifier} not implemented.`);
    }
  }
}

When creating an app, a single factory can be registered on the construct tree. Typically this is done through a base "app" class that handles this, such as cdktf-aws.App which makes sure to invoke PolyconFactory.register(this, new TerraformAWSFactory()). Then any applicable polycons will have a default concrete class and client. To override this resolution, extend a factory or create your own to specify your desired constructs.

export class MultiCloudFactory extends PolyconFactory {
  constructor() {
    this.awsFactory = new TerraformAWSFactory();
    this.gcpFactory = new TerraformGCPFactory();
  }

  resolve(qualifier: string, scope: Construct, id: string, props?: any) {
    if (qualifier === "std.Bucket") {
      // My favorite part of AWS is S3!
      return this.awsFactory.resolve(qualifier, scope, id, props);
    } else {
      // GCP is great for everything else!
      return this.gcpFactory.resolve(qualifier, scope, id, props);
    }
  }
}

Concepts​

Code and Processes​

Code is an immutable container of some code. As a primitive, it could be a file, directory, glob, inline text, or any other useful description. There are static methods on the Code class to create code from various sources such as Code.fromFile and Code.fromInline.

A Process contains Code, entrypoint, and is capable of adding Captures to interact with objects defined outside of their runtime. Conceptually you could think of Code as a library and Process as an application meant to be run.

Compute Platform​

Processes are run on compute platforms, which could be AWS Lambda, your local machine, Docker container, etc. Compute platforms must implement IComputePlatform to expose an API to allow clients to prepare them to run. The current API only includes a means to set environment variables but will be extended in the future to cover a larger common surface area.

// compute-platform.ts

export interface IComputePlatform {
    setEnvironment(name: string, value: string): void;
}

Captures​

When writing the runtime code for a process, you may want to interact with a construct defined during construction. Consider this code to download from a bucket:

// handler.ts

interface Captures {
  storage: BucketClient;
}

async function myHandler(event: QueueEvent, captures: Captures) {
  await captures.storage.download(event.records[0].id);
}

When creating the infrastructure for this we must ensure you have a BucketClient called storage created and passed to your entrypoint. To do this, create a Process with a Capture:

// app.ts

const app = new App();

const storage = new std.Bucket(app, "Storage");

const process: Process = {
  code: Code.fromFile(join(__dirname, "handler.ts")),
  entrypoint: "myHandler",
  captures: [
    Capture.client({
      name: "bucket",
      target: storage,
      methods: ["download"],
    }),
  ],
};

There are some static utility methods available through Capture to make this easy for custom clients or direct value serialization as well. The default runtime client for a polycon can be set by the factory defined for your app.

Full Example: WebGrep​

The example below shows the code for a construct that searches a graph of websites for a specific textual pattern. It starts from rootUrl and publishes the URLs of all the pages accessible from this URL to a topic called results. This construct is completely portable to any cloud, including the runtime.

You will notice that this construct uses a fictional library called pocix (stands for "portable cloud interface", an homage to [POSIX] which stands for "portable operating system interface"). This library exports a set of polycons which represent commonly used cloud resources such as Bucket, Topic, Queue and Function which WebGrep uses to implement its algorithm.

// app.ts

import * as std from "pocix";
import { Construct } from "constructs";

export interface WebGrepProps {
  readonly rootUrl: string;
  readonly pattern: string;
  readonly results: std.Topic;
}

export class WebGrep extends Construct {
  constructor(scope: Construct, id: string, props: WebGrepProps) {
    super(scope, id);
    const urlQueue = new std.Queue(this, "Queue");

    const process: Process = {
      code: Code.fromFile(join(__dirname, "worker.ts")),
      entrypoint: "handler",
      captures: [
        Capture.client({
          name: "results",
          target: props.results,
          methods: ["publish"],
        }),
        Capture.client({
          name: "queue",
          target: urlQueue,
          methods: ["push"],
        }),
        // Capture.direct injects the target directly into the process via serialization
        Capture.direct({
          name: "pattern",
          target: props.pattern
        })
      ],
    }

    const worker = new std.Function(this, "Function", {
      process,
    });

    // Ensures the Function listens to and gets items from the Queue
    urlQueue.addWorkerFunction(worker, { concurrency: 100 });

    // Tells the Queue to seed itself with an item during deployment
    urlQueue.enqueue(props.rootUrl);
  }
}
// worker.ts

import { QueueMessage, TopicClient, QueueClient } from "pocix-clients";

interface Captures {
  results: TopicClient;
  queue: QueueClient;
  pattern: string;
}

export async function handler(message: QueueMessage, captures: Captures) {
  const url = message.body;
  const html = await download_url(url);
  if (new RegExp(captures.PATTERN).test(html)) {
    await clients.results.publish(url);
  }

  // find all urls in html
  for (const url of await urlsFromPage(html)) {
    await clients.queue.push(url);
  }
}

End of README

Definitions​

Polymorphic​

Describes something that can take multiple shapes. In this sense, a polymorphic construct is a construct whose true implementation during runtime is determined by rules beyond its explicit instantiation.

Construction-Time​

Refers to the period of running the code where constructs themselves are defined. In AWS CDK terms, this is similar to synthesis but expanded to include the time before synthesis as well.

Process​

User-defined code running in some sort of compute system completely past construction time. For example: code running in an AWS Lambda function, inside a container, a VM, or a one-off job.

Captures​

A reference from a process to construction-time objects. This can manifest in many ways: A runtime client that interacts with a resource, environment variables containing specific resource data, or even a primitive value. Captures will be represented in your runtime code as "clients" that know how to interact with the resource.

Proposed Interfaces​

IComputePlatform​

This will likely be extended by polycons like IFunction (e.g. Lambda function) or IJob (e.g. CRON-based ECS task) where they intend to run a Process.

export interface IComputePlatform {
  setEnvironment(name: string, value: string): void;
}

ICaptureClient​

Has multiple responsibilities:

  1. Exposes Code (if needed) for consuming process to make available
  2. Has a method to make sure any changes needed for a compute platform are made for a given capture.
  3. Provides the actual string to be used in the process when constructing the shim entrypoint.
export interface ICaptureClient {
  readonly code: Code | undefined;

  bindToComputePlatform(capture: Capture, platform: IComputePlatform): void;

  renderCapture(capture: Capture): string;
}

Process​

Final name TBD

The description of code intended to be used in runtime.

export interface Process {
  readonly entrypoint: string;
  readonly captures: Capture[];
  readonly code: Code;
}