Serverless, Dynamo, GraphQL, Apollo Server, ElasticSearch, APIGateWay, Lambda, Graph(i)QL Playground, Kibana, TypeScript, NPM, and AWS

Gopi Krishna Kancharla

Published in

Think Special — Gopi K Kancharla

14 min readOct 17, 2019

⛹🏼 Let's make them work together

I am writing this while making breakfast 🍛 for my family. Its Dosa time! Today: Oct12, Frisco, TX.

🐜 ::: Introduction :::

While building a web application:

We need a database (Dynamo) for persistence
API for data retrieval and data storing,
Elastic Search to enable searching,
Data Streaming from the database to elastic-search,
GraphQL to query the data that is only needed,
APIGateway to aggregate the results from multiple services,
lambda for business logic executions written in TypeScript
A Serverless framework to enable all the above-mentioned Infrastructure to provision on a single command run.

Sounds Interesting? may or may not be yet. Thinking too many things to integrate and work together? Well, once we build a sample application you can feel surprised how fast we can build it. Hold on to the ground.

⚓️ ::: Let’s Dig :::

When a piece of software is installed on a machine that can bring the communication capabilities to it is called a server.

Serverless doesn’t mean that there is no server either from a Machine perspective or from a Software Perspective. So the meaning is: we literally don’t need to go to Cloud Consoles or use corresponding SDKs to create and manage the servers plus necessary infrastructure to run the applications. Serverless will take care of this part while you are focusing on the development.

This design will also bring additional complexities as sometimes third parties controlling the infrastructure, difficult debugging, too many dependencies, concurrency limits from cloud platforms, huge learning curve and resource skill sets, execution Startup latency, while, it also adds the benefits, like:

→ Loosely coupled systems: Example: Streaming is broken and for a short period of time, the search functionality may not be working while the rest of the application is working as designed. The search functionality can be fixed alone.

→ Horizontal scaling is possible with cloud platform constraints. Say, for example, AWS Lambda has a concurrency limit across the AWS account. You may observe DoS(Denial of service in your production). Still, we are able to do the horizontal scaling as the requests are growing up.

→ While we can build the application at the rocket speed, we can also come across the events where we need more time to debug that is not working as designed or expected.

→ No more OS rehydrations, VMs maintenance, traditional deployments.

Every tech stack system has both the sides and knowing these aspects will help to take the right decisions.

Architecture:

💻Let's create an example:

haigopi/serverless-aws-poc

You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…

github.com

Prerequisites: NPM & VSCode must be installed on your machine along with the below npm packages

🛠 npm i graphql-scalars class-transformer uuid-random
🛠 npm i g aws-sdk serverless

graphql-scalars → is for to use the graphql built-in datatypes.
class-transformer → is to use for Serialization and Deserialization
aws-sdk → is to connect and using the AWS services.
uuid-random → to generate the random ids.

To get the syntax highlights inside the VS Code, install the GraphQL, TypeScript Intellisense extensions as highlighted.

⛳Install TypeScript: TypeScript is a scripting language that is strongly typed and Object-Oriented ends with the .ts file extension. TypeScript will be compiled to JavaScript for execution. Uses the same syntax as JavaScript.

🛠 npm install -g typescript
🛠 tsc --version

Let's generate a typescript project using one of the many npm packages available. Make sure to have an internet connection as the packages will be pulled from npmjs repositories.

🛠 npx typescript-starter

Enter the required details and it few moments to install all the packages.

Change the directory to the folder(allibilli) and run the below command.

🛠 code .

Observe the files list:

👀 package.json → Similar to Gradle or Maven builds files. It has the dependencies, version, script subcommands to run supplied to “npm run” command. It helps to rebuild upon sharing to the public and by the public whoever running npm like package managers on their machines.

👀 tsconfig.json → Tells the root of the application and supplies the project compiler options to convert to the executable javascript files to the configurable destination folders.

👀 tslint.json → Static code analysis tool to check the code quality. This file can also force the developers to follow the coding standards.

You must need to understand the scripts section in the package.json

Default Scripts created for us.

Let's run the command:

🛠 npm run build

You will see the result as

🛠 F:\allibilli>npm run build> allibilli@1.0.0 build F:\experiments\medium\allibilli
> run-s clean && run-p build:*> allibilli@1.0.0 clean F:\experiments\medium\allibilli
> trash build test> allibilli@1.0.0 build:main F:\experiments\medium\allibilli
> tsc -p tsconfig.json> allibilli@1.0.0 build:module F:\experiments\medium\allibilli
> tsc -p tsconfig.module.json

If you carefully read the scripts section you will understand what exactly the command is doing.

Now let's imagine the use cases that we would like to do. For the website AlliBilli, all I would like to do is provide a facility for the users to read the existing newspapers in a nicer and easy way. For that, I would like to add the links and delete the links or modify the links. We are not building the UI as part of this article, but we are building the backend using the titled tech stack.

To do a few manual tests later, download and install the electron-based app called GraphQL Playground, which will be looking like as below screenshot when you open it. Note that the same application that is packaged in electron to run on the desktop, when we are running inside a browser is called GraphiQL.

Let's install the Serverless at first as a global NPM package so that the Serverless CLI will be available on your path. We will see down the line what it is and how does it affect our application.

🛠 npm i serverless -g

You can remove the libs and types that are created as examples to use under the src folder.

So, now we have a clean code repository to start with.

………🐘 Let's create a schema folder under src and create the following files.

👀 server.ts →

Coming to the point Apollo Server is an addition to the Node.js Http server designed to handle the GraphQL requests. It supports several data sources like REST, Database, etc

We will be instantiating the ApolloServer for the upcoming GraphQL based requests to handle. For this, lets install the apollo-server-lambda npm package.

🛠 npm install apollo-server-lambda

We will come back to this concept once we understand the next two files.

👀 types.graphql → GraphQL schema’s one of the basic components is Types which describes the object and its fields we will be retrieved or sent from/to the service.

Here is a sample type Link I just defined.

Apart from the Object definition, we also do have the Operation definitions as well. There are two types to discuss in this article named

🐰 Query → Defines the type of operation usually to fetch the data.

🐣 Mutation → Defines the type of operation usually to modify the data on the server-side. Though we can modify the data through Query type as well, it's not a best practice and a convention.

To make sure compile able to parse it, inform the TS upon adding a module.

Create a “@types” folder inside src having a file underneath, graphql.d.ts with the below content.

It's telling the compiler to use such files available as modules, like other npm packages.

👀 resolvers.ts → Resolvers are the actual functions that get executed and fetch the data from REST or from Database and then return the data in the type defined format. The client whoever querying for the data-set should be defined as part of these definitions. So, We can consider the types.graphql as a contract between the client and the server. Since we are purely working on the backend in this article, we will be using the GraphQL Playground as the client.

Now coming back to the server.ts → let's create the following code.

which has the mappings with the definitions and the corresponding resolvers to execute on a client request. Additional optional params like “introspection” and “playground” need to be set to true to enable the GraphQL Playground in production. This is not a good practice. They are enabled by default in development and disabled in production mode.

The final step is query the data from a database through a resolver. For this, we will write the resolvers and then we will be using Dynamo DB as our persistent store. We need to solve how to run dynamo locally for faster and quick development and connect to AWS Dynamo instance during production. We will see in a bit later how can we create the dynamo db infrastructure.

……… 🐘 As a next step, let’s create a “dynamodb” folder under src and create the following files. Make sure to install the Dynamo DB local setup.

🛠 npm install serverless-dynamodb-local

👀 dbClient.ts → The location where we do configure the connection parameters based on the environment.

Note the deal here, process.env is the environment specific properties. When you install a “dotenv” package and adding a couple of files like “.env.development” and “.env.production” those properties will be available in all your components. Major file talks about the way of creating the DynamoDBClient.

👀 dynamodb.yml → Place we will define the AWS CloudFormation type of resource and the dynamo table name and its attributes.

Note that the custom table name will be coming from serverless.yml which we will define shortly.

Now time to create the lambdas(resolvers) that can do the CRUD operations on the database.

Create the following files under a folder “operations”

createLink.ts → We will do DB insertions
deleteLink.ts → DB delete
getLink.ts → retrieves an event based on Id from the database
listLinks.ts → lists all the events available in the database table with a table scan. (This is, for example, don't do this in production)

createLink.ts

Similarly, we will do the Get/Delete/List as below in their respective files.

dbClient.get(params, callback)
dbClient.delete(params, callback)
dbClient.scan(params, callback)

When you observe the promisify in the above code, its a simple utility written for Node.js Promise to handle which runs the code asynchronously as below:

the callback is an argument function that will be called back on the result of a promise. Create a folder for handyUtils and then this file inside it.

Time for a couple of handlers to handle the requests and responses and communicating with the resolvers to make the database calls.

graphqlHandler → Handles the graphql queries.
processStreamsHandler → This handler is responsible to stream the data from the database to elastic search. We will write the resolver for this shortly.

create this file under src and name it as handler.ts

……… 🐘 Now let’s get on to Elastic Search.

🛠 npm i @elastic/elasticsearch

Create a folder “elasticSearch” and then the following files:

👀 elasticsearch.yml → Same like the cloud formation created for DynamoDB, Let's create for ES(ElasticSearch) like below.

👀 config.ts → This explains the configuration

👀 esConnect.ts →

👀 mappings.ts →

Create a subfolder under an elastic search called Stream and create the following files.

process.ts → The resolver that identifies the event happened on a Dynamo and hands over to the lambda function with the event.
index.ts → The place where we extract the data and stream to ES.

Let's compile once and see for errors:

🛠 npm run buildF:\experiments\medium\allibilli>npm run build> allibilli@1.0.0 build F:\experiments\medium\allibilli
> run-s clean && run-p build:*> allibilli@1.0.0 clean F:\experiments\medium\allibilli
> trash build test> allibilli@1.0.0 build:main F:\experiments\medium\allibilli
> tsc -p tsconfig.json> allibilli@1.0.0 build:module F:\experiments\medium\allibilli
> tsc -p tsconfig.module.jsonF:\experiments\medium\allibilli>

Great! Compile Success. ➰

Last Step to add the most important file: Serverless.yml. Per prerequisite, we installed serverless and aws-sdk as a global npm package. This brings the cli “sls” command to run.

You have to read the file several times to understand what is the meaning of each section.

custom → This section provides all the custom definitions that you would like to use in the project.

An interesting piece here is environment attribute:

environment: ${file(env.yml):${self:custom.stage}, file(env.yml):default}

So, create this file at the root of your project. env.yml

2. provider → this section helps to identify the cloud provider and its resource configurations. The following clouds are supported.

✔ AWS ✔ AZURE, ✔GCP, ✔ OPEN WHISK,✔KUBELESS, ✔ SPOTINST,✔FN, ✔CLOUDFLARE, ✔ALIBABA.

OpenWhisk is my favorite.💬 Being apart of these providers:

3. functions → are the section that maps your handlers. We have an echo handler for a test, graphql handler to handle the API calls over the API Gateway, Stream Processing for INSERTS and DELETES into DynamoDB.

** Make sure these handlers matched with the tsconfig.json outputs. in our case: output will be into build/main/

4. Resources → The section that explains the external files to be imported. (AWS Specific resources)

💎Without documentation, any product or library is just useless. So, Just run

npm run doc

You now can see the documentation generated and can browser all your properties, files, actions, mutations, queries etc from one place as below.

💦 Looks like we are all set, compile it, npm run build and on a successful compilation, its time to package, test and deploy.

Time to see the magic of serverless:

Let's pack the application:

🛠 sls package

It will error out as no plugins found as below:

Serverless Error ---------------------------------------Serverless plugin "serverless-webpack" not found. Make sure it's installed and listed in the "plugins" section of your serverless config file.

Add a webpack configuration:

and then install all the plugins needed per the plugins section in the serverless.yml and then package again.

🛠 npm install webpack serverless-webpack serverless-dynamodb-local serverless-plugin-offline-dynamodb-stream serverless-offline

Don’t forget to recompile. If not, customize the sections of the script in the package.json, as below.

"package": "npm run build && sls package","offline": "SET AWS_REGION=localhost && npm run compile && sls offline start --noTimeout","deploy": "SET AWS_REGION=us-east-1 && npm run build && sls deploy --stage prod",

Also, as part of the compilation, other files like .graphql will not be copied into the output directory. For this, we will use a simple utility called “cp”

🛠 npm install cpx

and add a script into the package.json.

“cp”: “cpx \”src/**/*.{yml,graphql}\” build/main”

and run this as part of the build process.

the complete script will be looking like this:

Now, let's run

🛠 npm run package

Great! So far! Its success and you should be able to see .serverless folder.

Now, run to start the application offline/local:

🛠 npm run offline

Then you may observe the following error:

Error --------------------------------------------------Error: spawn java ENOENT
      at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
      at onErrorNT (internal/child_process.js:415:16)
      at process._tickCallback (internal/process/next_tick.js:63:19)

That means its the time to install dynamo.

sls dynamodb install

You can now observe a “.dynamodb” folder created under your project.

run the offline command again and you will see the following on a successful start.

Dynamodb Local Started, Visit: http://localhost:15002/shell
Serverless: DynamoDB - created table default-events
Serverless: Starting Offline: default/us-east-1.Serverless: Routes for echo:
Serverless: POST /graphql
Serverless: GET /graphql
Serverless: POST /{apiVersion}/functions/link-news-default-echo/invocationsServerless: Routes for processStreams:
Serverless: POST /{apiVersion}/functions/link-news-default-processStreams/invocationsServerless: Offline [HTTP] listening on http://0.0.0.0:15001
Serverless: Enter "rp" to replay the last request

Observe the ports that we had been started at. Dynamo is running on 15002 while handlers are listening on the port 15001. Can you guess the name of the server that is started and listening for graphql requests on 15001?

You can also list the dynamodb shell running locally and paste the below script that I copied from the internet to scan the tables. The table name is given as per the serverless custom configuration table name.

stage: ${opt:stage, self:provider.stage}tableName: ${self:custom.stage}-events  <-- default-events

Open the browser and run localhost:15001/graphql and see the playground running.

You can also run this as a standalone app that you installed in the earlier step. GraphiQL Playground.

This is the one going to act as Client for us to our application.

Listing Links:

Adding Link:

Deleting Link, Observe the Id given from previous screen and name returns null.

You can also observe these events to ElasticSearch for adding and deleting documents from the DynamoDB.

Running offline?  true 15002
 ==> Event Recieved:  { Records:
   [ { eventID: '77a74f53-a93e-4906-9a94-665b4602522b',
       eventName: 'INSERT',
       eventVersion: '1.1',
       eventSource: 'aws:dynamodb',
       awsRegion: 'ddblocal',
       dynamodb: [Object] } ],
  NextShardIterator:
   '000|arn:aws:dynamodb:ddblocal:000000000000:table/default-events/stream/2019-10-14T01:15:19.762|c2hhcmRJZC0wMDAwMDAwMTU3MTAxNTcxOTgyMy0yOTJlN2VjNXwwMDAwMDAwMDAwMDAwMDAwMDAwMDJ8MDAwMDAwMDAwMDAwMDAwMDAxNTcxMDE1ODE4OTgy' } is Offline:  trueRunning offline?  true 15002
 ==> Event Recieved:  { Records:
   [ { eventID: 'fcfe9389-d556-4e24-9700-c1f0e53d57d1',
       eventName: 'REMOVE',
       eventVersion: '1.1',
       eventSource: 'aws:dynamodb',
       awsRegion: 'ddblocal',
       dynamodb: [Object] } ],
  NextShardIterator:
   '000|arn:aws:dynamodb:ddblocal:000000000000:table/default-events/stream/2019-10-14T01:15:19.762|c2hhcmRJZC0wMDAwMDAwMTU3MTAxNTcxOTgyMy0yOTJlN2VjNXwwMDAwMDAwMDAwMDAwMDAwMDAwMDN8MDAwMDAwMDAwMDAwMDAwMDAxNTcxMDE2MDQ2OTEz' } is Offline:  true

Since it is working as expected locally, let's deploy this to AWS with a single command. The whole Infrastructure will get created in a few MINUTES. I configured the AWS using aws configure command before running the below command.

F:\experiments\medium\allibilli>aws configure
AWS Access Key ID [****************WB2F]: DFGDFGDFGDGDFGDFGDFG
AWS Secret Access Key [****************p7O6]: DFGDFGDFGDFGFDGDFG
Default region name [us-east-2]:
Default output format [json]:
🛠 npm run deploy

Deal Done!

You will see the output as below with a complete infrastructure created.

Serverless: Packaging service...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
.....
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service link-news.zip file to S3 (2.78 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
......................................................
Serverless: Stack update finished...
Service Information
service: link-news
stage: prod
region: us-east-1
stack: link-news-prod
resources: 18
api keys:
  None
endpoints:
  POST - https://73b69myyl5.execute-api.us-east-1.amazonaws.com/prod/graphql
  GET - https://73b69myyl5.execute-api.us-east-1.amazonaws.com/prod/graphql
functions:
  echo: link-news-prod-echo
  processStreams: link-news-prod-processStreams
layers:
  None