Reference Architecture for AI

Support the project

Unknown — Tue, 08 Sep 2026 00:00:00 +0000

The patterns described on this site are not theory. They are extracted from Terraphim, a privacy-first AI assistant built across 52 Rust crates, and every one of those crates is open source.

The most valuable support is use and scrutiny.

Use it

terraphim.ai — the assistant itself: local-first, your data stays on your device.
terraphim.rs — the engine: 52 crates, each with a single responsibility and usable independently. Start with terraphim_automata for Aho-Corasick matching, terraphim_rolegraph for role-based knowledge graphs, or terraphim_persistence for the storage layer.

Contribute

Issues, benchmarks and architecture critique are worth more than money here. If a trade-off on this site looks wrong, say so — with evidence — at github.com/terraphim. Deployment guides and reproducible benchmarks are especially welcome.

If your organisation depends on this work, Terraphim Pro funds the open-source engine directly: connectors, an LLM proxy and agentic skills for enterprise deployment.

The Twelve Core Patterns

Unknown — Tue, 08 Sep 2026 00:00:00 +0000

The harness, not the model, determines agent success. A harness has three jobs — deciding what the model sees, enforcing what it can do, and ensuring it learns from its own history — and each job is served by four patterns.

Every pattern below is documented elsewhere on this site. This page is the index, not the explanation: each entry links to where the argument is actually made.

Context Architecture

What the model sees at each step. Context is a scarce resource: a 1M token window with 800K of noise performs worse than 200K with 150K of curated signal.

1. Structural Context Boundary

Put the boundary in configuration, not in a runtime permission check. A role has access to certain sources; the knowledge graph is built from those sources; the agent therefore cannot reach anything outside its role. Not a check that could be bypassed — an absence. The data was never loaded.

→ The Terraphim Engine, Layer 1

2. Deterministic Concept Matching

Resolve text to concepts with finite state automata rather than vector similarity. Aho-Corasick finds every match in O(n) time, with the same result every run. Probabilistic retrieval gives you a different context window each time you ask, which makes every downstream failure irreproducible.

→ The MCP-Native Agent Architecture, Layer 2

3. Structured Compaction

Compact before the window fills, not when it overflows. By 100K tokens a working context is largely noise: superseded search results, abandoned reasoning paths, old file reads. Adding capacity without compaction makes performance worse, not better.

→ Three ways 1M token windows fail

4. Priority-Weighted Instruction Injection

Instructions compete, and by default the loudest wins rather than the most important. The system prompt says one thing, the project file another, the README a third. Weight them explicitly and re-inject the high-priority ones, rather than trusting position in a transcript.

→ Three ways 1M token windows fail

Execution Guardrails

What the model can and cannot do. Risk-tiered, machine-readable, and enforced by the system rather than requested in a prompt.

5. Multi-Tenant Agent Isolation

Agents sharing a protocol layer must not share state. Each agent keeps its own role configuration and its own knowledge graph, discovering the others' tools without discovering their data.

→ Pattern 1: Multi-Tenant Agent Isolation

6. Deterministic Replay

An agent run you cannot replay is an agent run you cannot debug. Record the inputs and the decisions such that the same run can be reconstructed exactly — which is only possible if the layers underneath are themselves deterministic.

→ Pattern 2: Deterministic Replay

7. Human-in-the-Loop for Destructive Operations

Classify tools by effect, not by name. Read-only operations run freely; destructive ones stop for approval. The protocol enforces the distinction, so an agent cannot talk its way past it.

→ Pattern 3: Human-in-the-Loop for Destructive Operations

8. Break-Glass

Every guardrail needs a deliberate, logged way through it. A guardrail with no override gets disabled wholesale the first time it blocks something urgent; an override that is not recorded is indistinguishable from the guardrail not existing.

→ Harness 3: Break-Glass

Memory Infrastructure

Ensuring the model learns from its own history. Without it, agents repeat solved problems indefinitely and no amount of context helps.

9. Novelty Filter

Distinguish output that is new from output that merely looks new. An agent that cannot tell repetition from progress will re-report the same finding until someone stops reading.

→ Harness 1: Novelty Filter

10. Calculated Health

Derive a health signal from observable metrics rather than asking the agent how it is doing. Self-assessment is the least reliable input available; commit recency, unapplied lessons and stall time are not.

→ Harness 2: Calculated Health

11. Lesson Window

Capture corrections at the moment of failure and apply them within a bounded window. Lessons with no expiry become noise; lessons never applied were never lessons.

→ Harness 4: Lesson Window

12. Drift Detection

Long-running agents optimise for local coherence and lose the original objective. Ask periodically, and mechanically: do we actually have the thing we set out to build? The question has to be asked by the harness, because an agent that has drifted cannot detect its own drift.

→ Three ways 1M token windows fail · The nine-step orchestration arc

On this list

These twelve are the patterns this site currently makes an argument for, grouped by the three jobs of an agent harness. The grouping is deliberate: a pattern that does not serve one of those three jobs probably belongs in a different architecture.

The list is not a claim to completeness. It is what is documented, and the honest test of any addition is whether there is an article behind it.

The Terraphim Engine: Six Layers, Fifty-Two Crates

Unknown — Tue, 08 Sep 2026 00:00:00 +0000

The diagram on the front page of this site is not an illustration. Every box in it is a published Rust crate you can cargo add today, read the documentation for, and use without taking the rest of the system. This article explains what sits behind each box, and why the decomposition is the interesting part.

Why decomposition is the architecture

Most agent frameworks ship as a single dependency. You take the orchestrator, the memory layer, the retrieval stack and the tool protocol together, or you take none of it. That is convenient at first and expensive later: when one layer misbehaves you cannot isolate it, and when you want to replace one you replace all of them.

The pattern this site argues for is the opposite. Draw the boundaries first, make each one independently useful, and let the seams stay visible. Terraphim is built that way — 52 crates across six layers, each with a single responsibility — which makes it a reasonable worked example rather than a product pitch.

The test of a boundary is whether you can use one side without the other. Here you can: terraphim_automata is a text-matching engine that happens to be used by a knowledge graph; terraphim_persistence is a storage abstraction that does not know what it is storing.

The six layers

Terraphim organises its crates into a dependency graph with six levels. Lower layers do not know the higher ones exist.

Layer	Concern	Representative crates
6	User interfaces	`terraphim_agent`, `terraphim-cli`, `terraphim_server`
5	Orchestration	`terraphim_orchestrator`, `terraphim_kg_orchestration`, `terraphim_symphony`
4	Agent system	`terraphim_spawner`, `terraphim_router`, `terraphim_agent_supervisor`
3	Service layer	`terraphim_service`, `terraphim_middleware`, `haystack_*`
2	Core engine	`terraphim_automata`, `terraphim_rolegraph`, `terraphim_persistence`
1	Types and configuration	`terraphim_types`, `terraphim_config`, `terraphim_settings`

That ordering matters more than the crate count. An agent system that cannot say which layer a failure came from is a system you cannot debug in production.

Layer 1: types and configuration

terraphim_types is the shared vocabulary — the structures every other crate agrees on. terraphim_config holds role definitions, haystacks and LLM routing; terraphim_settings handles runtime preferences.

Putting roles in configuration rather than code is what makes the context boundary structural. A role has access to certain haystacks; the knowledge graph is built from those haystacks; therefore the agent cannot reach information outside its role. That is not a permission check at runtime that could be bypassed — it is an absence. The data was never loaded.

Layer 2: the core engine

This is the layer the front-page diagram mostly depicts.

terraphim_automata is the matching engine: Aho-Corasick finite state automata that match thousands of patterns simultaneously in O(n) time, where n is the length of the text. Not "usually fast" — bounded, and the same bound every time. It compiles to WebAssembly, which is why the same matcher runs in a browser as on a server.

terraphim_rolegraph is the knowledge graph. It is deliberately not a general-purpose graph database. It maps search roles to domain-specific graph views, and it exists to do one thing quickly: turn text into concepts, deterministically. Terraphim reports knowledge-graph inference in the 5-10 nanosecond range and a 15 MB memory footprint, with no GPU — figures worth checking against your own workload, but the shape of the claim follows from the design rather than from optimisation tricks.

terraphim_persistence provides the Persistable trait and DeviceStorage backends across memory, SQLite and redb. Storage is an interface here, not an assumption, which is what lets the same engine run on a laptop and at the edge.

Layer 3: the service layer

terraphim_middleware searches haystacks — pluggable data-source backends. This is the boundary the front-page diagram labels Document Input. A haystack might be a local folder, a repository, or a connector to something else; the layers above do not change when you add one.

terraphim_service handles requests and responses for the core, which keeps request handling out of the engine itself.

Layers 4 and 5: agents and orchestration

Above the service layer sit the agent system (terraphim_spawner, terraphim_router, terraphim_agent_supervisor) and orchestration (terraphim_orchestrator, terraphim_kg_orchestration, terraphim_symphony).

The distinction is worth holding onto. The agent system is concerned with individual agents — spawning them, routing to them, supervising them. Orchestration is concerned with what happens between them. Collapsing those two into one "agent framework" is the usual mistake, and it is why so many systems cannot tell a stuck agent from a stuck workflow.

The MCP-Native Agent Architecture goes into the protocol surface these layers expose, and why treating MCP as an architectural primitive rather than a transport changes what the system can guarantee.

Layer 6: the interfaces

terraphim_agent is the interactive REPL with session search and learning capture. terraphim-cli is the automation-shaped counterpart, with JSON output for scripting. terraphim_server provides the REST API.

Three interfaces over one engine, none of them privileged. That is the payoff of the layering: the CLI is not a thin wrapper around the server, and the server is not a special case. They are peers over the same core.

What to take from this

You do not need to adopt Terraphim to use the pattern. The transferable parts are:

Draw layers that a failure can be attributed to. If you cannot name the layer, you cannot fix the bug.
Make each boundary independently useful. A crate nobody would use on its own is usually not a boundary, it is a split file.
Put the context boundary in configuration, not in runtime checks. Structural impossibility beats a permission test.
Keep the expensive guarantees low. Determinism in the matching engine is worth more than determinism bolted onto the orchestrator.

Documentation

Every crate named above is published and documented:

terraphim.rs — the engine: all 52 crates, grouped by layer, with the dependency graph and installation instructions.
docs.rs — per-crate API documentation, generated from source. Start with terraphim_automata for matching, terraphim_rolegraph for the knowledge graph, or terraphim_persistence for storage.
docs.terraphim.ai — the full documentation site: guides, deployment and integration patterns.
terraphim.ai — the assistant built on all of it, if you would rather see the system working before reading its parts.

Installation is a single command, or cargo install terraphim_agent if you would prefer to start from the REPL.

The MCP-Native Agent Architecture

Unknown — Mon, 07 Sep 2026 00:00:00 +0000

A reference architecture for production AI systems, drawn from five years building Terraphim — a privacy-first AI assistant with ~50 Rust crates, MCP-native interfaces, and role-based multi-agent orchestration.

Harness Engineering for Self-Improving Agents

Unknown — Sun, 06 Sep 2026 00:00:00 +0000

Many AI agent systems plateau after deployment. They execute tasks, but the surrounding system may not measure whether they are getting better, repeating old work, or normalizing stalled execution. This article proposes a reference pattern for agent harnesses: supervised feedback loops that make agent behavior observable, measurable, and easier to improve.

Turning Open Source project into Product with Redis Enterprise

Unknown — Fri, 19 Aug 2022 00:00:00 +0000

Turning Open Source project into Product with Redis Enterprise

Overview

Background

History

Last year, my reference project, "The Pattern", was the hackathon winner 2021 and got a bit of publicity and, in total, seven forks. But as with many open source projects, it is now stale. Time to revive "The Pattern" with new features and GitHub sponsors or Patreon patrons to help and inspire developers and creatives. In return, it's common to provide sponsor-only features and articles. Nevertheless, how can we do it with a large Redis-based machine learning pipeline?

This article will introduce a simple first step: for GitHub sponsors, we start with offering persistent storage of preferences: I have a simple flask POST API which adds nodes into the user's preference storage - a simple Redis set per user. And it will be a foundation to build other sponsor-only features. For now, let's cover the basics:

Overall architecture overview

flowchart LR
    id1(User) --> flask_login(Flask Login API)--> github(GitHub OAuth2)
    github-->flask_callback(Flask API callback)-->GitHubGraphQL(GitHub GraphQL)

Add Github oauth2 to Rest API

There are a number of API's that GitHub offers to help developers, but the GitHub Authentication API is one of the most popular. This API allows you to log in to GitHub using your username and password, or OAuth token.

A login button with a standard OIDC/OAuth2 dance is one of the most common ways for a user to authenticate to an API. Below is code taken from this gist and is very common for OAuth2 flows:

import os 
client_id = os.getenv('GITHUB_CLIENT_ID')
client_secret = os.getenv('GITHUB_SECRET')
@app.route('/', methods=['GET', 'POST'])
def index():
    url = 'https://github.com/login/oauth/authorize'
    params = {
        'client_id': client_id,

        'scope': 'read:user,read:email',
        'state': str(uuid4().hex),
        'allow_signup': 'true'
    }
    url = furl(url).set(params)
    return redirect(str(url), 302)

where GITHUB_CLIENT_ID and GITHUB_SECRET are client/secret GitHub Oauth2 apps. Register for following process on GitHub

On callback:
- Fetch username, email, and other profile information- in case we need to contact them
- get the status of the user - if they are sponsor of the organisation
  - if they are sponsor
  - add the username to the set of sponsors in Redis

org_name="applied-knowledge-systems"
@app.route('/oauth2/callback')
def oauth2_callback():

    code = request.args.get('code')
    access_token_url = 'https://github.com/login/oauth/access_token'
    payload = {
        'client_id': client_id,
        'client_secret': client_secret,
        'code': code,
        # 'redirect_uri':
        'state': str(uuid4().hex)
    }
    r = requests.post(access_token_url, json=payload, headers={'Accept': 'application/json'})
    access_token = json.loads(r.text).get('access_token')
    print(access_token)
    
    access_user_url = 'https://api.github.com/user'
    response = requests.get(access_user_url, headers={'Authorization': 'token ' + access_token})
    data=response.json()
    user_email=data["email"]
    user_login=data["login"]
    user_id=data["id"]
    # response=redirect(url_for('login',next=redirect_url()))
    # response.set_cookie('user_id', str(user_id))
    # response.set_cookie('user_login', str(user_login))
    # return response
    query = """
        {
        viewer {
            sponsorshipsAsSponsor(first: 100) {
            nodes {
                sponsorable {
                ... on User {
                    id
                    email
                    url
                }
                ... on Organization {
                    id
                    email
                    name
                    url
                }
                }
                tier {
                id
                name
                monthlyPriceInDollars
                monthlyPriceInCents
                }
            }
            }
        }
        }
        """
    response_graphql = requests.post('https://api.github.com/graphql', json={'query': query}, headers={'Authorization': 'token ' + access_token})
    response_graphql_data=response_graphql.json()["data"]
    if isinstance (response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"], list):
      if response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"][0]["sponsorable"]["name"]==org_name:
          # if user is a sponsor of Applied Knowledge System add them to set of sponsors
         redis_client.sadd(f'sponsors:{org_name}',user_id)
    # if RedisJSON enabled:
    # redis_client.json().set(f"user_details:{user_id}", '$', {
    #     'email': user_email,
    #     'id': user_id,
    #     'user_login': user_login,
    #     'graphql': response_graphql_data,
    # }) 
    #if not
    redis_client.hset(f"user_details:{user_id}", mapping={
        'email': user_email,
        'id': user_id,
        'user_login': user_login
    })
    return jsonify({
        'status': 'success',
        'email': user_email,
        'id': user_id,
        'user_login': user_login
    })

The API we are using for our sponsor-only feature is straightforward:

@app.route('/exclude', methods=['POST','GET'])
def mark_node():
    if request.method == 'POST':
        if 'id' in request.json:
            node_id=request.json['id']
    else:
        if 'id' in request.args:
            node_id=request.args.get('id')
    user_id = session.get('user_id')
    log(f"Got user {user_id} from session")
    if not user_id:
        user_id = request.cookies.get('user_id')
        log(f"Got user {user_id} from cookie")
    redis_client.sadd("user:%s:mnodes" % user_id,node_id)
    response = jsonify(message=f"Finished {node_id} and {user_id}")
    return response

And the only purpose of this API is to mark nodes as unimportant for the given user by adding nodes to RedisSet, and those nodes will be excluded from search API output. So far, everything was pretty standard: basic flask API and GitHub Social login flow. Now let's add Redis Enterprise and synchronise sponsors preferences.

Add Redis Enterprise

Why not use Redis Enterprise directly for everything? The project is memory-heavy, with a lot of data and machine learning inside Redis. This allows to achieve state-of-the-art performance, but it also takes over 120 GB RAM (or as much RAM as you can give it), and 128 GB Redis Enterprise instance will exceed my budget for open-source project. Obviously if there will be enough sponsors we can move more functionality into Redis Enterprise, but for that we need to finish building basic blocks. Register on Redis.com cloud and create a database with the subscription.

Take a note host, port and password for Redis Enterprise and create docker enviroment file:

cat .env.gears 
REDISENT_PWD="123"
REDISENT_PORT="13444"
REDISENT_HOST="hostname.cloud.redislabs.com"

and create a docker compose with section passing .env.gears. Mine looks like this:

  redisgraph:
    image: redislabs/redismod
    container_name: redisgears
    hostname: redisgears
    env_file:
      - ./.env.gears
    ports:
      - 127.0.0.1:9001:6379

Synchronize Redis OSS to Redis Enterprise using RedisGears

Synchronize all user preferences

First flow: We will be using RedisGears to synchronize all preferences with Redis Enterprise

flowchart LR
    id1(User Preferences Redis OSS) --> redis_gears1(Redis Gears)--> redise(Redis Enterprise)

If you are new to RedisGears, there is a pattern rgsync that covers exacly this use case, but I already have RedisGears, so I am going to build it step by step:

# gears_sync_preferences.py
rconn=None

def connecttoRedisEnterise():
    import redis 
    import os 
    log(str(os.environ))
    # Get environment variables

    HOST = os.getenv('REDISENT_HOST')
    PASSWORD = os.getenv('REDISENT_PWD')
    PORT = os.getenv('REDISENT_PORT')

    redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)
    return redis_client

def sync_users(record):
    global rconn
    if not rconn:
        rconn=connecttoRedisEnterise()
    # Uncomment logs to check 
    # log(str(record['key']))
    # log(str(record['value']))

    rconn.hset(record['key'],mapping=record['value'])

gb = GB()
gb.foreach(sync_users)
gb.count()
gb.run('user_details:*')

this is a "batch" mode for RedisGears, which is easier to debug than streams. Install gears-cli](https://github.com/RedisGears/gears-cli) with pip install gears-cli run above script:

gears-cli run --host 127.0.0.1 --port 9001 gears_sync_preferences.py --requirements req_sync.txt

where req_sync.txt

redis==3.5.3

This RedisGears will copy all user's profiles into RedisEnterprise. Now let us add sponsors:

# gears_sync_sponsors.py
rconn=None

def remove_prefix(text, prefix):
    return text[text.startswith(prefix) and len(prefix):]

def connecttoRedisEnterise():
    import redis 
    import os 
    log(str(os.environ))
    # Get environment variables

    HOST = os.getenv('REDISENT_HOST')
    PASSWORD = os.getenv('REDISENT_PWD')
    PORT = os.getenv('REDISENT_PORT')
    log(HOST)
    log(PORT)
    log(PASSWORD)
    redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD)
    return redis_client



def sync_sponsors(record):
    global rconn
    if not rconn:
        rconn=connecttoRedisEnterise()

    log(str(record['key']))
    values=execute('SMEMBERS',record['key'])
    log(str(values))
    for each_value in values:    
        rconn.sadd(record['key'],each_value)

gb = GB('KeysReader')
gb.foreach(sync_sponsors)
gb.count()
gb.run('user:*')

But this one will sync all user's preferences, but we only need sponsors - let us add another feature of RedisGears - filter:

rconn=None

def remove_prefix(text, prefix):
    return text[text.startswith(prefix) and len(prefix):]

def connecttoRedisEnterise():
    import redis 
    import os 
    log(str(os.environ))
    # Get environment variables

    HOST = os.getenv('REDISENT_HOST')
    PASSWORD = os.getenv('REDISENT_PWD')
    PORT = os.getenv('REDISENT_PORT')

    redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)
    return redis_client

def filter_sponsors(record):
    org_name="applied-knowledge-systems"
    user_id = remove_prefix(record['key'],'user:')
    sponsor=execute('SISMEMBER',f'sponsors:{org_name}',user_id)
    return bool(sponsor==True)

def sync_sponsors(record):
    global rconn
    if not rconn:
        rconn=connecttoRedisEnterise()

    log(str(record['key']))
    values=execute('SMEMBERS',record['key'])
    log(str(values))
    for each_value in values:    
        rconn.sadd(record['key'],each_value)

gb = GB()
gb.filter(filter_sponsors)
gb.foreach(sync_sponsors)
gb.count()
gb.run('user:*')

Then we are going to use Key miss events from Redis Gears to fetch data for all users:

flowchart LR
    redise(Redis Enterprise)
    redis_gears2(Redis Gears)--key miss--->redise
    redis_gears2-->redisOSS[Redis OSS]

and it's very easy, right from key miss example:


def fetch_data(r):
    key = r['key']
    global rconn
    if not rconn:
        rconn=connecttoRedisEnterise()
    values=rconn.smembers(record['key'])
    log(str(values))
    for each_value in values:    
        execute('SADD',record['key'],each_value)

GB().foreach(fetch_data).register(prefix='user:*', commands=['smember'],eventTypes=['keymiss'], mode="async_local")

There is one more option - to turn fetch_data into the async call, by wrapping it into async/await, but Redis Enterprise is fairly fast, and I don't think it's worth adding an async call in this case. For curiosity, see the example code in The Pattern repository.

Conclusion

In this article, we walked through steps on how to create sponsor-specific "nanoservices" using RedisOSS, RedisGears and Redis Enterprise. This allows us to leverage the best of all worlds open source Redis, high availability and persistence with Redis Enterprise and RedisGears as the glue which holds everything together.

This post is in collaboration with Redis.

References

Benchmarks for BERT Large Question Answering inference for RedisAI and RedisGears

Unknown — Thu, 16 Jun 2022 00:00:00 +0000

Summary of the article

This article will explore the challenges and opportunities of deploying a large BERT Question Answering Transformer model(bert-large-uncased-whole-word-masking-finetuned-squad) from inside Huggingface, where RedisGears and RedisAI perform heavy lifting while leveraging in-memory datastore Redis.

Why do we need RedisAI?

In data science load, you want to load high-performance hardware as close to 100% as possible
In user-facing load, you want to be able to distribute the load evenly, so it never reaches 100%, and client-facing servers can perform additional functions
In data science, you prefer re-calculate results
In a client-facing application, you prefer to cache results of calculation and fetch data from the cache as fast as possible to drive a seamless customer experience

Some numbers for inspiration and why to read this article:

python3 transformers_plain_bert_qa.py 
airborne transmission of respiratory infections is the lack of established methods for the detection of airborne respiratory microorganisms
10.351818372 seconds

time curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearch

real	0m0.747s
user	0m0.004s
sys	0m0.000s

Background

BERT Question Answering inference works where the ML model selects an answer from the given text. In other words, BERT QA "thinks" through the following: "What is the answer from the text, assuming the answer to the question exists within the paragraph selected."

So it's important to select text potentially containing an answer. A typical pattern is to use Wikipedia data to build Open Domain Question Answering.

Our QA system is a medical domain-specific question/answering pipeline, hence we need a first pipeline that turns data into a knowledge graph. This NLP pipeline is available at Redis LaunchPad, is fully open source, and is described in a previous article. Here is a 5 minute video describing it, and below you will find an architectural overview:

BERT Question Answering pipeline and API

In the BERT QA pipeline (or in any other modern NLP inference task), there are two steps:

Tokenize text - turn text into numbers
Run the inference - large matrix multiplication

With Redis, we have the opportunity to pre-compute everything and store it in memory, but how do we do it? Unlike with the summarization ML learning task, the question is not known in advance, so we can't pre-compute all possible answers. However, we can pre-tokenize all potential answers (i.e. all paragraphs in the dataset) using RedisGears:

def parse_sentence(record):
    import redisAI
    import numpy as np
    global tokenizer
    if not tokenizer:
        tokenizer=loadTokeniser()
    hash_tag="{%s}" % hashtag()

    for idx, value in sorted(record['value'].items(), key=lambda item: int(item[0])):
        tokens = tokenizer.encode(value, add_special_tokens=False, max_length=511, truncation=True, return_tensors="np")
        tokens = np.append(tokens,tokenizer.sep_token_id).astype(np.int64)
        tensor=redisAI.createTensorFromBlob('INT64', tokens.shape, tokens.tobytes())

        key_prefix='sentence:'
        sentence_key=remove_prefix(record['key'],key_prefix)
        token_key = f"tokenized:bert:qa:{sentence_key}:{idx}"
        redisAI.setTensorInKey(token_key, tensor)
        execute('SADD',f'processed_docs_stage3_tokenized{hash_tag}', token_key)

See the full code on GitHub.

Then for each Redis Cluster shard, we pre-load the BERT QA model by downloading, exporting it into torchscript, then loading it into each shard:

def load_bert():
    model_file = 'traced_bert_qa.pt'

    with open(model_file, 'rb') as f:
        model = f.read()
    startup_nodes = [{"host": "127.0.0.1", "port": "30001"}, {"host": "127.0.0.1", "port":"30002"}, {"host":"127.0.0.1", "port":"30003"}]
    cc = ClusterClient(startup_nodes = startup_nodes)
    hash_tags = cc.execute_command("RG.PYEXECUTE",  "gb = GB('ShardsIDReader').map(lambda x:hashtag()).run()")[0]
    print(hash_tags)
    for hash_tag in hash_tags:
        print("Loading model bert-qa{%s}" %hash_tag.decode('utf-8'))
        cc.modelset('bert-qa{%s}' %hash_tag.decode('utf-8'), 'TORCH', 'CPU', model)
        print(cc.infoget('bert-qa{%s}' %hash_tag.decode('utf-8')))

The full code is available on GitHub.

And when a question comes from the user, we tokenize and append the question to the list of potential answers before running the RedisAI model:

    token_key = f"tokenized:bert:qa:{sentence_key}"
    # encode question
    input_ids_question = tokenizer.encode(question, add_special_tokens=True, truncation=True, return_tensors="np")
    t=redisAI.getTensorFromKey(token_key)
    input_ids_context=to_np(t,np.int64)
    # merge (append) with potential answer, context - is pre-tokenized paragraph
    input_ids = np.append(input_ids_question,input_ids_context)
    attention_mask = np.array([[1]*len(input_ids)])
    input_idss=np.array([input_ids])
    num_seg_a=input_ids_question.shape[1]
    num_seg_b=input_ids_context.shape[0]
    token_type_ids = np.array([0]*num_seg_a + [1]*num_seg_b)
    # create actual model runner for RedisAI
    modelRunner = redisAI.createModelRunner(f'bert-qa{hash_tag}')
    # make sure all types are correct
    input_idss_ts=redisAI.createTensorFromBlob('INT64', input_idss.shape, input_idss.tobytes())
    attention_mask_ts=redisAI.createTensorFromBlob('INT64', attention_mask.shape, attention_mask.tobytes())
    token_type_ids_ts=redisAI.createTensorFromBlob('INT64', token_type_ids.shape, token_type_ids.tobytes())
    redisAI.modelRunnerAddInput(modelRunner, 'input_ids', input_idss_ts)
    redisAI.modelRunnerAddInput(modelRunner, 'attention_mask', attention_mask_ts)
    redisAI.modelRunnerAddInput(modelRunner, 'token_type_ids', token_type_ids_ts)
    redisAI.modelRunnerAddOutput(modelRunner, 'answer_start_scores')
    redisAI.modelRunnerAddOutput(modelRunner, 'answer_end_scores')
    # run RedisAI model runner
    res = await redisAI.modelRunnerRunAsync(modelRunner)
    answer_start_scores=to_np(res[0],np.float32)
    answer_end_scores = to_np(res[1],np.float32)
    answer_start = np.argmax(answer_start_scores)
    answer_end = np.argmax(answer_end_scores) + 1
    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end],skip_special_tokens = True))
    log("Answer "+str(answer))
    return answer

Checkout the full code, available on GitHub.

The process for making a BERT QA API call looks like this:

Here I use two cool features of RedisGears: capturing events on key miss and using async/await to run RedisAI on each shard without locking the primary thread - so that Redis Cluster can continue to serve other customers. For benchmarks, caching responses from RedisAI is disabled. If you are getting response times in nanoseconds on the second call rather then milliseconds, check to make sure the line linked above is commented out.

Running the Benchmark

Pre-requisites for running the benchmark:

Assuming you are running Debian or Ubuntu and have Docker and docker-compose installed (or can create a virtual environment via conda), run the following commands:

git clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git
cd the-pattern
./bootstrap_benchmark.sh

The above commands should end with a curl call to the qasearch API, since Redis caching is disabled for the benchmark.

Next, invoke curl like this:

time curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearch

Expect the following output, or something similar based on your runtime environment:

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Sun, 29 May 2022 12:05:39 GMT
Content-Type: application/json
Content-Length: 2120
Connection: keep-alive

{"links":[{"created_at":"2002","rank":13,"source":"C0001486","target":"C0152083"}],"results":[{"answer":"adenovirus","sentence":"The medium of 40 T150 flasks of adenovirus transducer dec CAR CHO cells yielded 0 5 1 my of purified msCEACAM1a 1 4 protein","sentencekey":"sentence:PMC125375.xml:{mG}:202","title":"Crystal structure of murine sCEACAM1a[1,4]: a coronavirus receptor in the CEA family"}] OUTPUT_REDUCTED}

I modified the output of API for the benchmark to return results from all shards - even if the answer is empty, in the run above five shards return answers, overall API call response under second with all additional hops to search in RedisGraph.

I modified the output of the API for the benchmark to return results from all shards - even if the answer is empty. In the run above five shards return answers. The overall API call response takes less than one second with all additional hops to search in RedisGraph!

Deep Dive into the Benchmark

Let's dig deeper into what's happening under the hood:

You should have a sentence key with shard id, which you get by looking at the "Cache key" from docker logs -f rgcluster. In my setup the cache key is, "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults". If you think it looks like a function call it's because it is a function call. It is triggered if the key isn't present in the Redis Cluster, which for the benchmark will be every time since if you remember we disabled caching the output.

One more thing to figure out from the logs is the port of the shard corresponding to the hashtag, also known as the shard id. It is the text found in betweeen the curly brackets – looks like {6fd} above. The same will be in the output for the export_load script. In my case the cache key was found in "30012.log", so my port is 30012.

Next I run the following command:

redis-cli -c -p 300012 -h 127.0.0.1 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"

and then run the benchmark:

redis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"
====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======
  10 requests completed in 0.04 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

10.00% <= 41 milliseconds
100.00% <= 41 milliseconds
238.10 requests per second

If you are wondering, -n = number of times. In this case we run the benchmark 10 times. You can also add:

– csv if you want to output in CSV format

– precision 3 if you want more decimals in the ms

More information about the benchmarking tool can be found on the redis.io Benchmarks page.

if you don't have redis-utils installed locally, you can use Docker as follows:

docker exec -it rgcluster /bin/bash
redis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"
====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======
  10 requests completed in 1.75 seconds
  50 parallel clients
  99 bytes payload
  keep alive: 1
  host configuration "save":
  host configuration "appendonly": no
  multi-thread: no

Latency by percentile distribution:
0.000% <= 243.711 milliseconds (cumulative count 1)
50.000% <= 987.135 milliseconds (cumulative count 5)
75.000% <= 1577.983 milliseconds (cumulative count 8)
87.500% <= 1662.975 milliseconds (cumulative count 9)
93.750% <= 1744.895 milliseconds (cumulative count 10)
100.000% <= 1744.895 milliseconds (cumulative count 10)

Cumulative distribution of latencies:
0.000% <= 0.103 milliseconds (cumulative count 0)
10.000% <= 244.223 milliseconds (cumulative count 1)
20.000% <= 409.343 milliseconds (cumulative count 2)
30.000% <= 575.487 milliseconds (cumulative count 3)
40.000% <= 821.247 milliseconds (cumulative count 4)
50.000% <= 987.135 milliseconds (cumulative count 5)
60.000% <= 1157.119 milliseconds (cumulative count 6)
70.000% <= 1497.087 milliseconds (cumulative count 7)
80.000% <= 1577.983 milliseconds (cumulative count 8)
90.000% <= 1662.975 milliseconds (cumulative count 9)
100.000% <= 1744.895 milliseconds (cumulative count 10)

Summary:
  throughput summary: 5.73 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
     1067.296   243.584   987.135  1744.895  1744.895  1744.895

The platform only has 20 articles and 8 Redis nodes (4 masters + 4 slaves), so relevance would be wrong and it doesn't need a lot of memory.

AI.INFO

Now let's check how long our RedisAI model runs on the {6fd} shard:

127.0.0.1:30012> AI.INFO bert-qa{6fd}
 1) "key"
 2) "bert-qa{6fd}"
 3) "type"
 4) "MODEL"
 5) "backend"
 6) "TORCH"
 7) "device"
 8) "CPU"
 9) "tag"
10) ""
11) "duration"
12) (integer) 8928136
13) "samples"
14) (integer) 58
15) "calls"
16) (integer) 58
17) "errors"
18) (integer) 0

bert-qa{6fd} is the key of the actual (very large) model saved. The AI.INFO command gives us a cumulative duration of 8928136 microseconds and 58 calls, which is approximately 153 milliseconds per call.

Let's double-check to make sure that's right by resetting the stats and then re-runnning the benchmark.

First, reset the stats:

127.0.0.1:30012> AI.INFO bert-qa{6fd} RESETSTAT
OK
127.0.0.1:30012> AI.INFO bert-qa{6fd}
 1) "key"
 2) "bert-qa{6fd}"
 3) "type"
 4) "MODEL"
 5) "backend"
 6) "TORCH"
 7) "device"
 8) "CPU"
 9) "tag"
10) ""
11) "duration"
12) (integer) 0
13) "samples"
14) (integer) 0
15) "calls"
16) (integer) 0
17) "errors"
18) (integer) 0

Then, re-run the benchmark:

redis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"
====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======
  10 requests completed in 1.78 seconds
  50 parallel clients
  99 bytes payload
  keep alive: 1
  host configuration "save":
  host configuration "appendonly": no
  multi-thread: no

Latency by percentile distribution:
0.000% <= 188.927 milliseconds (cumulative count 1)
50.000% <= 995.839 milliseconds (cumulative count 5)
75.000% <= 1606.655 milliseconds (cumulative count 8)
87.500% <= 1692.671 milliseconds (cumulative count 9)
93.750% <= 1779.711 milliseconds (cumulative count 10)
100.000% <= 1779.711 milliseconds (cumulative count 10)

Cumulative distribution of latencies:
0.000% <= 0.103 milliseconds (cumulative count 0)
10.000% <= 189.183 milliseconds (cumulative count 1)
20.000% <= 392.191 milliseconds (cumulative count 2)
30.000% <= 540.159 milliseconds (cumulative count 3)
40.000% <= 896.511 milliseconds (cumulative count 4)
50.000% <= 996.351 milliseconds (cumulative count 5)
60.000% <= 1260.543 milliseconds (cumulative count 6)
70.000% <= 1456.127 milliseconds (cumulative count 7)
80.000% <= 1606.655 milliseconds (cumulative count 8)
90.000% <= 1692.671 milliseconds (cumulative count 9)
100.000% <= 1779.711 milliseconds (cumulative count 10)

Summary:
  throughput summary: 5.62 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
     1080.454   188.800   995.839  1779.711  1779.711  1779.711

Now check the stats again:

AI.INFO bert-qa{6fd}
 1) "key"
 2) "bert-qa{6fd}"
 3) "type"
 4) "MODEL"
 5) "backend"
 6) "TORCH"
 7) "device"
 8) "CPU"
 9) "tag"
10) ""
11) "duration"
12) (integer) 1767749
13) "samples"
14) (integer) 20
15) "calls"
16) (integer) 20
17) "errors"
18) (integer) 0

Now we get 88387.45 microseconds per call ~0.088387 seconds, which is pretty fast! Also, considering we started with 10 seconds per call, I think the benefits of using RedisAI in combination with RedisGears are pretty obvious. However, the trade-off is high memory usage.

There are many ways to optimize this deployment. For example, you can add a FP16 quantization and ONNX runtime. If you would like to try that, this script will be a good starting point.

Using Grafana to monitor RedisGears throughput, CPU, and Memory usage

Thanks to the contribution of Mikhail Volkov, we can now observe RedisGears and RedisGraph throughput and memory consumption using Grafana. When you cloned repository it started Graphana Docker, which has pre-build templates to monitor RedisCluster, including RedisGears and RedisAI, and Graph - which is Redis with RedisGraph. "The Pattern" dashboard provides an overview, with all the key benchmark metrics you care about:

This post is in collaboration with Redis.

Capability Map (2020-2022)

Unknown — Thu, 16 Jun 2022 00:00:00 +0000

Introduction

There are tools for advanced analytics, including free ones from Google and Kaggle.

There are well-known and validated deployment architectures for applications and the cloud.

Yet the number of practical applications is still tiny, and they retained niche implementations. While the benefits of AI are clear, there are still many gaps in AI architecture that need to be filled. For example, there is a gap between analytical tools and verified architectures for real-time deployments. This gap often stems from a lack of specific reference architectures and patterns, demonstrating the trade-offs between technologies, libraries, and tools.

Let's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments.

Three things need to be in place to build an AI product:

AI Product itself
Core Capabilities required to build AI/ML product
Enabling capabilities

I will use The Pattern, my Build on Redis Hackathon prize-winning open source project to illustrate how the capabilities below can be implemented and invite you to contribute or donate. The diagrams below are clickable.

AI Product

Core capabilities for AI/ML

flowchart LR; 
subgraph " "
  data_intake(Data Acquisition);
  data_preprocessing(Data Preprocessing);
  data_validation(Data Validation);
  data_streaming(Data Streaming);
  ai_ml_pipe(AI ML Pipeline 
 Model Training/Test/Validation);
  kg(Knowledge Graph);
  ML_inference(ML Inference);
  interaction(Interaction Layer
Voice/VR/AR/Meta);
  click data_intake "/docs/intake/" "Data Acquisition"
  click ML_inference "/docs/bert-qa-benchmarking/" "BERT Large Question Answering"
  click kg "/docs/nlp/#what-is-a-knowledge-graph" "Knowledge Graph"
  click data_preprocessing "/docs/nlp/#redisgears-for-nlp-pre-processing" "Data pre-processing"
  click data_streaming "/docs/nlp/#goal" "Data Streaming"
  click ai_ml_pipe "/docs/nlp/#overall-architecture-overview-components-diagram" "Overall ML pipeline"
end

Enabling Capabilities

flowchart LR;
subgraph 2 [Enabling Capabilities]
  subgraph " "
D1[Data Governance]
D2[Data Quality Management]
D3[Metadata Management]
click D3 "/docs/metadata/" "Metadata Management"
end
subgraph " "
  id2[ML performance and bias monitoring];
  id3[Application Performance Monitoring];
  id4[Hardware Performance Monitoring];
  click id3 "/docs/bert-qa-benchmarking/#running-the-benchmark" "Running Benchmarks"
  click id4 "/docs/bert-qa-benchmarking/#using-grafana-to-monitor-redisgears-throughput-cpu-and-memory-usage" "CPU and Memory Benchmarks"
end
subgraph " "
  id5[DevOps: Continuous Integration/Continuous Deployment];
  c1[Configuration management]
  c2[Collaboration and knowledge management tooling]
  c3[Change Management]
end 
  subgraph 3 [Self Serving Infrastructure]
    s1[Computing Infrastructure]
    s2[Serving infrastructure]
  end
  end
  style 2 fill:#485fc754,stroke:#333,stroke-width:4px;

Announcing Reference Architecture for AI

Unknown — Thu, 16 Jun 2022 00:00:00 +0000

There are tools for advanced analytics, including free ones from Google and Kaggle.

There are well-known and validated deployment architectures for applications and the cloud.

Let's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments. Three things need to be in place to build an AI product:

AI Product itself
Core Capabilities required to build AI/ML product
Enabling capabilities

I will use The Pattern, my [“Build on Redis” Hackathon prize-winning open source](https://github.com/applied-knowledge-systems/the-pattern) project, to illustrate how the capabilities below can be implemented and invite you to contribute or donate.

We launch in two full-featured articles - NLP ML pipeline for turning unstructured JSON text into a knowledge graph and fresh off the press Benchmarks for BERT Large Question Answering inference for RedisAI and RedisGears with Grafana Dashboards by Mikhail Volkov

Building a Pipeline for Natural Language Processing using RedisGears

Unknown — Sat, 28 May 2022 00:00:00 +0000

Goal

Disclaimer originally published in collaboration with Ajeet Raina on Developer.Redis.Com

In this tutorial, you will learn how to build a pipeline for Natural Language Processing(NLP) using RedisGears. For this demonstration, we will be leveraging the Kaggle CORD19 datasets. The implementation is designed to avoid running out of memory, leveraging Redis Cluster and RedisGears, where the use of RedisGears allows for processing data on storage without the need to move data in and out of the Redis Cluster—using Redis Cluster as data fabric. Redis Cluster allows for horizontal scalability up to 1000 nodes, and together with RedisGears, provides a distributed system where data science/ML engineers can focus on processing steps, without the worry of writing tons of scaffoldings for distributed calculations.

This project was built with the aim to make it easier for other people to contribute and build better information and knowledge management products.

Why data scientists uses RedisGears?

RedisGears have enormous potential, particularly for text processing—you can process your data “on data” without needing to move them in and out of memory. Summary of the important points:

In Memory storage (Horizontally scalable if it’s Redis cluster)
Processing of data (on data) without need to move in and out
Gears - like Spark on Hadoop, process data intelligently on storage(in-memory) without need to move data in and out
Redis in cluster mode with RedisGears and python enabled takes 20 MB RAM. Think how much more data you can shuffle into your laptop or server.

What is a knowledge graph?

Today, we live in the world of new systems that operate not just files, folders, or web pages, but entities with their properties and relationships between them, organized into hierarchies of classes and categories. These systems are used everywhere from the military-industrial complex to our everyday lives. Palantir, Primer, and other data companies enable massive intelligence and counterintelligence projects in military and security forces, Quid and RecordedFuture enable competitive analytics, Bottlenose and similar enterprises enable online reputation analytics. Microsoft Graph enables new kinds of productivity apps for the enterprises, Google Knowledge Graph and Microsoft’s Satori enable everyday search queries, and together with Amazon Information Graph they power corresponding AI assistants by enabling them to answer questions about the world facts

All these (and many other more specialized) systems are used in different domains, but all of them use Knowledge Graphs as their foundation.

Knowledge graphs are one of the best ways to connect and make sense out of information from different data sources, following the motto of one of the vendors— “It’s about things not strings”.

Knowledge Graph consists of thesaurus, taxonomy and ontology. In this pipeline I assume knowledge is captured in medical metathesaurus UMLS and concepts in text are related if they are part of the same sentence, therefore concept become node, their relationship becomes edge:

Concepts have CUI (Concept Unique Identifiers) and those will be primary keys in nodes, linked to UMLS thesaurus. For example, if you search, “How does temperature and humidity affect the transmission of 2019-nCoV?” on the demo website http://thepattern.digital/ and move slider to 1996, there is an edge-connecting transmission (C5190195) and birth (C5195639) and the part of sentence matched, “the rate of transmission to an infant born to,” from the report titled, “Afebrile Pneumonia in infants.”

RedisGears for NLP pre-processing

Overall Architecture Overview (Components Diagram)

Intake step - is very simple put all JSON records into RedisCluster, then NLP pipeline starts processing all records, code is here.

How does the NLP pipeline steps fit into RedisGears?

For each record — detect language (discard non English), it’s filter
Map paragraphs into a sentence — flatmap
Sentences spellchecker — it’s map
Save sentences into hash — processor

Step 1. Pre-requisite

Ensure that you install virtualenv in your system

Step 2. Clone the repository

 git clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git
 cd the-pattern

Step 3. Bring up the application

 docker-compose -f docker-compose.dev.yml up --build -d

Step 4. Apply cluster configuration settings

You can deploy PyTorch and spacy to run on RedisGears.

 bash post_start_dev.sh

For Data science-focused deployment, RedisCluster should be in HA mode with at least one slave for each master. One need to change a few default parameters for rgcluster to accommodate the size of PyTorch and spacy libraries (each over 1GB zipped), gist with settings.

Step 5. Create or activate Python virtual environment

 cd ./the-pattern-platform/

Step 6. Create new environment

You can create it via

 conda create -n pattern_env python=3.8

Alternatively, you can activate by using the below CLI:

 source ~/venv_cord19/bin/activate #or create new venv
 pip install -r requirements.txt

Step 7. Run pipeline

 bash cluster_pipeline.sh

Step 8. Validating the functionality of the NLP pipeline

Wait for a bit and then check:

Verifying Redis Graph populated:

 redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (n:entity) RETURN count(n) as entity_count" 
 redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (e:entity)-[r]->(t:entity) RETURN count(r) as edge_count"

Checking API responds:

 curl -i -H "Content-Type: application/json" -X POST -d '{"search":"How does temperature and humidity affect the transmission of 2019-nCoV"}'      
 http://localhost:8080/gsearch

Walkthrough

While RedisGears allows to deploy and run Machine Learning libraries like spacy and BERT transformers, the solution above uses simpler approach:

 gb = GB('KeysReader')
 gb.filter(filter_language)
 gb.flatmap(parse_paragraphs)
 gb.map(spellcheck_sentences)
 gb.foreach(save_sentences)
 gb.count()
 gb.register('paragraphs:*',keyTypes=['string','hash'], mode="async_local")

This is the overall pipeline: those 7 lines allow you to run logic in a distributed cluster or on a single machine using all available CPUs - no changes required until you need to scale over more 1000 nodes. I use KeysReader registered for namespace paragraphs for all strings or hashes. My pipeline would need to run in async mode. For data scientists, I would recommend using gb.run to make sure gears function work and it will run in batch mode and then change it to register - to capture new data. By default, functions will return output, hence the need for count() - to prevent fetching the whole dataset back to the command issuing machine (90 GB for Cord19).

Overall pre-processing is a straightforward - full code is here.

Things to keep in mind:

Node process can only save locally - we don't move data, anything you want to save should have hashtag, for example to add to the set of processed_docs:

 execute('SADD','processed_docs_{%s}' % hashtag(),article_id)

Loading external libraries into the computational threat, for example, symspell requires additional dictionaries and needs two steps to load:

 """
 load symspell and relevant dictionaries
 """
 sym_spell=None 

 def load_symspell():
  import pkg_resources
  from symspellpy import SymSpell, Verbosity
  sym_spell = SymSpell(max_dictionary_edit_distance=1, prefix_length=7)
  dictionary_path = pkg_resources.resource_filename(
      "symspellpy", "frequency_dictionary_en_82_765.txt")
  bigram_path = pkg_resources.resource_filename(
      "symspellpy", "frequency_bigramdictionary_en_243_342.txt")
  # term_index is the column of the term and count_index is the
  # column of the term frequency
  sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)
  sym_spell.load_bigram_dictionary(bigram_path, term_index=0, count_index=2)
  return sym_spell

Scispacy is a great library and data science tool, but after a few iterations with deploying it I ended up reading data model documentation for UMLS Methathesaurus and decided to build Aho-Corasick automata directly from UMLS data. (MRXW_ENG.RRF contains all terms form for English mapped to CUI). Aho-Corasick allowed me to match incoming sentences into pairs of nodes (concepts from the medical dictionary) and present sentences as edges in a graph, Gears related code is simple:

 bg = GearsBuilder('KeysReader')
 bg.foreach(process_item)
 bg.count()
 bg.register('sentence:*',  mode="async_local",onRegistered=OnRegisteredAutomata)

OnRegisteredAutomata will perform similarly to symspell example above except it will download pre-build Aho-Corasick automata (30Mb). Aho-Corasick is a very fast matcher and allows to perform >900 Mb text per second even on commodity laptop, RedisGears cluster makes a very smooth distribution of data and ML model and matching using available CPU and Memory. Full matcher code.

Output of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams. But I decided to use streams and handcraft the population of the RedisGraph database, which will be focus of the next blog post.

Output of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams, while this demo uses streams and populates RedisGraph database with nodes and edges calculating rank of each.

Call to action

We took OCR scans in JSON format and turned them into Knowledge Graph, demonstrating how you can traditional Semantic Network/OWL/Methathesaurus technique based on Unified Medical Language System. Redis Ecosystem offers a lot to the data science community, and can take place at the core of Kaggle notebooks, ML frameworks and make deployment and distribution of data more enjoyable. The success of our industry depends on how our tools work together — regardless of whether they are engineering, data science, machine learning and organisational or architectural.

With the collaboration of RedisLabs and community, the full pipeline code is available via https://github.com/applied-knowledge-systems/the-pattern-platform. In case, you want to try it locally, then you can find a Docker Launch script in the root of the repository along with short quickstart guide. PR and suggestions are welcome. The overall goal of the project is to allow other to build their more interesting pipeline on top of it.

References

Metadata Management

Unknown — Sat, 14 May 2022 00:00:00 +0000

In CORD 19 dataset mentioned in Data Acquisition Metadata stored in the separate csv file from the source data. Here simple script to parse date/times and attach it to JSON/XML files

Contribution Guidelines

Unknown — Wed, 15 Dec 2021 00:00:00 +0000

General guidelines for contributing to the project.

The Pattern: Machine Learning Natural Language Processing meets VR/AR

Unknown — Mon, 31 Aug 2020 00:00:00 +0000

To fight ever-increasing complexity, "The Pattern" projects help find relevant knowledge using Artificial Intelligence and novel UX elements, all powered by Redis - a new generation real-time data fabric turned into knowledge fabric

Overall repository for CORD19 medical NLP pipeline, API and UI, design and architecture.

Demo Video:

Demo Server (no persistance): https://thepattern.digital/

The challenge

The medical profession put a lot of effort into collaboration, starting from Latin as a common language to industry-wide thesauruses like UMLS. However, if full of scandals where publications in a prestigious journal would be retracted, and the World Health Organisation would change its policy advice based on the article. I think "paper claiming that eating a bat-like Pokémon sparked the spread of COVID-19" takes a prize. One would say that editors in those journals don't do their job, and while it may seem true, I would say they had no chance: with a number of publications about COVID (SARS-V) passing 300+ per day, we need better tools to navigate via such flow of information. When exploring science or engineering topics, I look at the diversity of the opinion, not the variety of the same cluster of words or the same thought. I want to avoid confirmation bias. I want to find articles relevant to the same concept, not necessarily the ones which have similar words. My focus is to build a natural language processing pipeline capable of handling a large number of documents and concepts, incorporating System 1 AI (fast, intuitive reasoning) and System 2 (high-level reasoning) and then present knowledge in a modern VR/AR visualisation. Search or rather information exploration should be spatial, preferably in VR (memory palace, see Theatre of Giulio Camillo). A force-directed graph is a path towards it, where visuals are assisted by text — relevant text pops up on the connection and where people explore the concepts and then dig deeper into the text. The purpose of the pipeline is that knowledge should be reusable and shareable.

Community

Join our community on Discord or post on GitHub Discussions](https://github.com/applied-knowledge-systems/the-pattern/discussions)

Data Acquisition

Unknown — Mon, 31 Aug 2020 00:00:00 +0000

For the Reference Architecture for AI, we used Kaggle Cord19 dataset, "COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 1,000,000 scholarly articles, including over 400,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease."

Ingest documents

Example script parses documents taking out body_text and saves under paragraphs in Redis cluster.

Reference Architecture for AI

Support the project

Use it

Contribute

Sponsor

The Twelve Core Patterns

Context Architecture

1. Structural Context Boundary

2. Deterministic Concept Matching

3. Structured Compaction

4. Priority-Weighted Instruction Injection

Execution Guardrails

5. Multi-Tenant Agent Isolation

6. Deterministic Replay

7. Human-in-the-Loop for Destructive Operations

8. Break-Glass

Memory Infrastructure

9. Novelty Filter

10. Calculated Health

11. Lesson Window

12. Drift Detection

On this list

The Terraphim Engine: Six Layers, Fifty-Two Crates

Why decomposition is the architecture

The six layers

Layer 1: types and configuration

Layer 2: the core engine

Layer 3: the service layer

Layers 4 and 5: agents and orchestration

Layer 6: the interfaces

What to take from this

Documentation

The MCP-Native Agent Architecture

Harness Engineering for Self-Improving Agents

Turning Open Source project into Product with Redis Enterprise

Turning Open Source project into Product with Redis Enterprise

Overview

Background

History

Plan sponsor only features

Overall architecture overview

Add Github oauth2 to Rest API

Add Redis Enterprise

Synchronize Redis OSS to Redis Enterprise using RedisGears

Synchronize all user preferences

Fetch sponsor's preferences back to Redis OSS from Redis Enterprise

Conclusion

References

Benchmarks for BERT Large Question Answering inference for RedisAI and RedisGears

Summary of the article

Why do we need RedisAI?

Background

BERT Question Answering pipeline and API

Running the Benchmark

Deep Dive into the Benchmark

AI.INFO

Using Grafana to monitor RedisGears throughput, CPU, and Memory usage

Capability Map (2020-2022)

Introduction

AI Product

Core capabilities for AI/ML

Enabling Capabilities

Announcing Reference Architecture for AI

Building a Pipeline for Natural Language Processing using RedisGears

Goal

Why data scientists uses RedisGears?

What is a knowledge graph?

RedisGears for NLP pre-processing

Overall Architecture Overview (Components Diagram)

How does the NLP pipeline steps fit into RedisGears?

Step 1. Pre-requisite

Step 2. Clone the repository

Step 3. Bring up the application

Step 4. Apply cluster configuration settings

Step 5. Create or activate Python virtual environment

Step 6. Create new environment

Step 7. Run pipeline

Step 8. Validating the functionality of the NLP pipeline

Verifying Redis Graph populated:

Checking API responds: