Fluvio - secret sauce for scalability

[joepio]

By @AlexMikhalev, replaces #461

Getting Started - warm up and machine to machine communications

minikube status
fluvio profile current
fluvio cluster check

Code quality "Hello World", for discussion and not merge - large change and it's combined "Hello World" for Fluvio Rust library and examples/basic.rs
First modify basic example to emit new article on successful save:
record()
Second add consume_record, where we are:

1. Reading fluvio stream
2. parsing key into subject (key for Atomic Data Server)
3. Parsing value into resource
4. If successful saving into local (in memory in this example storage)

Start with

1. cargo run --example async_std_consumer --all-features -- produce_record
2. Check that data is available in topic fluvio consume atomic-data -B
3. cargo run --example async_std_consumer --all-features -- consume_record

That closes Machine 2 Machine communication discussion: #425 . Obviously we can now extend commit hook to produce record into fluvio topic in the same way as you added tantivy indexing.

Now cool stuff

We have: JSON_AD importer
#390

Atomizer
#434

Wasm plugins
#73

discussions and roadmap items. As it happens Fluvio supports HTTP (and other) connectors, which allow us to run Smartmodule (WASM plugins) on "the edge" - before data arrives into fluvio topic.
So we can build a highly scalable Atomizer with an option to pre-process data "on-fetch" and "on-save". If it sounds like dream come true, this is because it is ;-)

Lets start by building fetcher for JSON-AD from reference-architecture.ai
Follow:
https://www.fluvio.io/smartmodules/transform/array-map/
I created array-map-ref-arch-ad don't use underscores in name of connector or topic - fluvio doesn't like it in many places.

cd array-map-ref-arch-ad
cargo build --release

fluvio smart-module list 
(should be empty)
fluvio smart-module create array-map-ref-arch-ad --wasm-file ./target/wasm32-unknown-unknown/release/array_map_ref_arch_ad.wasm
fluvio smart-module list
  NAME                   STATUS             SIZE   
  array-map-ref-arch-ad  SmartModuleStatus  156905

Now start http connector locally:

docker run -d --name="my-http" \
-v"$HOME/.fluvio/config:/home/fluvio/.fluvio/config" \
-t infinyon/fluvio-connect-http:latest \
-- \
--endpoint="https://reference-architecture.ai/json-ad" \
--fluvio-topic="atomic-data" \
--interval=10s --arraymap="array-map-ref-arch-ad"

Http connector can be deployed into Infinyon cloud or run locally, since I use minikube it can't fetch externally so it fails. #FIXME

Connector parameters can be specified reference_architecture.yml and registered on cloud with

fluvio connector create --config=./reference_architecture.yml

[AlexMikhalev]

Not relevant any more.