# A Conceptual Architecture for a Filesystem to SQS Loader

DevFeed: [A Conceptual Architecture for a Filesystem to SQS Loader](<https://devfeed.tech/articles/a-conceptual-architecture-for-a-filesystem-to-sqs-loader-28185.md>)

Original publisher: [Read original article](<http://fuzzyblog.io/blog/lambda/2020/06/12/a-conceptual-architecture-for-a-filesytem-to-sqs-loader.html>)

Author: Fuzzygroup

Published: 2020-06-12T00:00:00Z

Content type: opinion

Language: en

Sources: [Scott Johnson](<https://devfeed.tech/sources/scott-johnson.md>)

Topics: [Amazon Simple Queue Service (SQS)](<https://devfeed.tech/topics/amazon-simple-queue-service-sqs.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Docker](<https://devfeed.tech/topics/docker.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [container](<https://devfeed.tech/tags/container.md>), [files](<https://devfeed.tech/tags/files.md>), [json](<https://devfeed.tech/tags/json.md>), [lambda](<https://devfeed.tech/tags/lambda.md>), [messages](<https://devfeed.tech/tags/messages.md>), [queue](<https://devfeed.tech/tags/queue.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [sqs](<https://devfeed.tech/tags/sqs.md>)

## AI overview

The article proposes a conceptual architecture for loading JSON messages from a filesystem into Amazon SQS. It considers replacing a single Rust file processor with AWS Lambda functions, a network-accessible filesystem, and Redis-based tracking for files already processed.

## Source excerpt

I have an interesting technical situation facing me: A docker container which runs persistently and writes JSON files to a directory Each JSON file contains short form messages with JSON encoding along a variety of metadata fields Each file is idempotent i.e. once it exists in the directory will never be written to again Each file is named for a timestamp Each file needs to: Have ridiculous, non-JSON comments removed from the beginning Be parsed into json There is roughly 6000 of these JSON things coming in per second Have only a selection of the JSON internally be selected Be batched into a collection up to the maximum size the SQS supports and then committed to SQS Be processed only once Note: SQS is the AWS "Simple Queue Service". A queue is a specialized data structure which hands data off for processing by other tasks. My initial thinking to handle this was to use Rust to write a high performance file processor. This appealed to me: New technical challenge; I don't know Rust Manly - writing fast code is a conveniently macho challenge Simple architecture - one box, one process watching a directory and ripping through files There are, naturally, problems here: The Rust program represents a SPOF or "Single Point of Failure"; things break This architecture is simple but making it not be a SPOF means the architecture gets complex We don't actually even know if a single Rust executable is fast enough to handle all the data Rust is complex and even simple things take dramatically more code than a scripting language like Ruby or Python. There isn't anything wrong with Rust (I really like the language but learning it is non-trivial). Thinking through all these issues as well as learning, by chance, that the runtime execution period for AWS lambda serverless functions had increased from 5 minutes to 15 minutes made me think in terms of a different architecture focused on using lambdas. Note: A lambda is a self contained bit of code that you give over to AWS to manage on