I think the larger part implied is the design will be crappy, because the problem was unexplored

I want to build a shared postgres db with hundreds of small apps (OLTP) accessing shared tables using a RLS model model against well defined tables.

What are other limitations and mitigations folks have used or encountered to support stability and security? Things like

  - Query timeouts to prevent noisy neighbors
  - connection pooling (e.g. pgbouncer) also for noisy neighbors
  - client schema compatibility (e.g. some applications running older versions, have certain assumptions about the schema that may change over time)

If you have people running crappy SQL SELECT, it can be a problem. statement-timeout[0] is your friend here. You still have to be on watch, and teach your users not to write crappy SQL.

You can also limit it by creating read-only replica's and making SELECT's happen on the replica. We don't usually bother, since 99% of our users are employees of ours, we can teach them to not be stupid. Since their usage doesn't change much over time, we can usually just hand them a SQL query and say: here run this instead.

Most of our employees don't even know they have SQL access, it's not like we force people to learn SQL to get their job done. Because of RLS and views, the ones that do SQL don't have to know much SQL, even if they do happen to use it. SELECT * from employees; gets them access to basically all the employee info they could want, but only to the employees they have access to. If you are a manager with 10 people, your select returns only your 10 people.

The payroll staff runs the same query and gets all of the employees they handle payroll for. Since our payroll is done inside of PostgreSQL(thanks plpython[1]), we can do some crazy access control stuff that most systems would never even dream about. Whenever new auditors come in and see that our payroll staff is limited to seeing only the info they need to do payroll, and only for their subset of employees they actually pay, they are awestruck.

The random vendors that can't be taught, we usually hand them a nightly SQLite dump instead. I.e let them pay the CPU cost of their crappy SQL.

Around client schema compatibility. This happens with other models too(API, etc). It's not unique to PG or SQL Databases. You have to plan for it. Since most all of our users interact with views and not with the actual underlying tables, it's not usually that big of a deal. In the extreme cases, where we can't just keep around a view for them, we have to help them along(sometimes kicking and screaming) into a new version.

0: https://www.postgresql.org/docs/current/runtime-config-clien...

1: https://www.postgresql.org/docs/current/plpython.html

github repository here https://github.com/run-llama/liteparse

it's Friday, and I'm avoiding doing work

Ha! Whelp, it is my weekend.

I notice that the python versions and typescript versions are pretty different. Python is sort of class based, with python magic decorators

    class MyWorkflow(Workflow):
        @step
        async def start(self, ctx: Context, ev: StartEvent) -> MyEvent:
            num_runs = await ctx.get("num_runs", default=0)

whereas TS is sort of builder/function based

    import { createWorkflow } from "@llamaindex/workflow-core";
    
    const convertEvent = workflowEvent();
    
    const workflow = createWorkflow();
    
    workflow.handle([startEvent], (start) => {
      return convertEvent.with(Number.parseInt(start.data, 10));

Is there reason for this? });

yea good callout -- python workflows came first, and while we could have directly translated these, the ergonomics around classes in python are not exactly what JS/TS devs expect.

So instead, the goal was to capture the spirit of event-driven workflows, and implement them in a more TS-native way and improve the dev-ux for those developers. This means it might be harder to jump between the two, but I'd argue most people are not doing that anyways.

Agree, thanks for the link. I was wondering what actually changed. The resource links and elicitation look like useful functionality.

I've been curious to see the process for selecting relevant context from a long conversation. has anyone reverse engineered what that looks like? how is the conversion history pruned, and how is the latest state of a file represented?

We didn't look into that workflow closely, but you can reproduce our work (code in GitHub) and potentially find some insights!

We plan to continue investigating how it works (+ optimize the models and prompts using TensorZero).

I recently did something similar. Using uv workspaces, I used the uv CLI's dependency graph to analyze the dependency tree then conditionally trigger CI workflows for affected projects. I wish there was a better way to access the uv dependency worktree other than parsing the `tree` like output

I use Github Actions triggers to pass flags to a monorepo dagger script to build and test the affected components. For example, if a commit touches the front- and back ends, rebuild both. If it only touches the front end, run integration tests using the latest backend without rebuilding it.

edit: spell out GHA

Yea this definitely makes sense for smaller monorepos. For us, we ended up writing our own dependency graph parser to figure out what tests to run (which is easy enough with a single language like python honestly)

Was bazel an option?

We used pants initially (which I believe is similar to bazel). And indeed the dependency graphing it does was very helpful, but other parts of the tool motivated us to create something more bespoke and debuggable (we were only using like 20% or less of the features pants offers)

GHA - GitHub Actions, right?

I agree! I hope uv introduces more tools for monorepos or refines the workspaces concept.

I saw workspaces require all dependencies to agree with eachother, which isn't quite possible in our repo

This is cool and something that I’ve wanted, but I don’t see hot listings requirements inline foregoes the need for a lock file to maintain reproducibility. What about version ranges? Versions of transitive dependencies?

Likely uses https://docs.astral.sh/uv/guides/scripts/#improving-reproduc...

Mobile 4g USB sticks you can usually rotate your IP address by reconnecting. I tried on a pi, it was inconsistent. This was just with some random test mobile plan from rando carrier renting off Verizon I think