WIP: cli

Author: closedLoop

.env

@@ -64,3 +64,7 @@ target/
 
 #Ipython Notebook
 .ipynb_checkpoints
+.env
+
+repomix-output.txt
+openapi.json

.gitignore

@@ -1,147 +1,271 @@
-# Infactory - Development
+# Infactory SDK
 
-This repository contains the back-end execution and query environment for the Infactory platform.
+The Infactory SDK provides simple and powerful interfaces to work with your data and AI services. It allows you to connect your data sources, create query programs, and publish them as APIs, all from your terminal or Python code.
 
-![Latest_Release](https://img.shields.io/github/v/release/infactory-io/dev)
-![Tests](https://github.com/infactory-io/dev/actions/workflows/tests.yml/badge.svg)
-![Coverage](./docs/images/coverage.svg)
-[![Language](https://img.shields.io/badge/python-3.11%20%7C%203.12-blue)](/#)
+## Installation
 
-## Running Locally
+```bash
+pip install infactory
+```
 
-### Using Docker
+## Getting Started
 
-First, ensure your environment is setup (take a look at the `Contributing: Environment Setup` below):
+### Setting your API Key
+
+Before using the SDK, you need to set your API key. You can either set it as an environment variable or use the CLI login command:
 
 ```bash
-# create the multi-platform build environment / driver
-docker buildx create --name multiarch --use
-docker buildx inspect --bootstrap
+# Set the API key as an environment variable
+export NF_API_KEY=your_api_key_here
 
-# build it
-docker compose build --builder multiarch --push
+# Or use the CLI login command
+nf login
 ```
 
-Then:
+## CLI Examples
+
+The Infactory CLI (`nf`) provides a set of commands to interact with the Infactory platform.
+
+### 1. Login to Infactory
 
 ```bash
-docker compose up
+$ nf login
+Enter your API key: ********************
+API key saved successfully!
 ```
 
-### Using Minikube (Kubernetes)
+### 2. Connect a Postgres datasource
 
-The Infactory framework leverages [Kubernetes](https://kubernetes.io) to deploy, scale and manage micro services, databases and other containerized applications within the framework. To leverage Kubernetes and run the framework locally, please install both [kubectl](https://kubernetes.io/docs/reference/kubectl/) and [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Farm64%2Fstable%2Fbinary+download) (which runs in Docker).
+```bash
+$ nf datasource create --name "Product Database" --type postgres --project-id my-project-id
+Datasource created successfully!
+ID: ds-abc123
+Name: Product Database
+Type: postgres
+
+$ nf datasource configure ds-abc123 --uri "postgresql://username:password@host:port/database"
+Datasource configured successfully!
+```
 
-If you're on a Mac, you can install `kubectl` and `minikube` via Homebrew:
+### 3. Subscribe to jobs for the data source
 
 ```bash
-brew install kubectl
-brew install minikube
+$ nf jobs subscribe --datasource-id ds-abc123
+Subscribing to jobs for datasource ds-abc123...
+[2025-03-25 14:30:21] Job j-123456 started: Connecting to PostgreSQL database
+[2025-03-25 14:30:22] Job j-123456 progress: Successfully connected to database
+[2025-03-25 14:30:25] Job j-123456 progress: Analyzing table structure
+[2025-03-25 14:30:30] Job j-123456 progress: Found 12 tables with 450,000 rows total
+[2025-03-25 14:30:45] Job j-123456 completed: Database connection established and schema analyzed
 ```
 
-Some folks on Macs report having issues installing the ARM tailored versions using `homebrew`. If that's what you run into, check out [these instructions](https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/).
-
-Now, do this:
+### 4. Generate a query program
 
 ```bash
-cd deploy
-chmod u+x ship_it.sh
-sh ship_it.sh
+$ nf query generate --dataline-id dl-456def --name "Monthly Sales by Region"
+Query program generation started...
+Analyzing data structure...
+Generating query program...
+Query program created successfully!
+ID: qp-789ghi
+Name: Monthly Sales by Region
 ```
 
-If you're on the `dev` branch or another branch, this script will deploy the Infactory framework to your local Minikube Kubernetes cluster. If on `staging` or `main`, the deployments will be pushed to the AWS Kubernetes cluster running in EKS, either on the `staging` or `prod` namespaces (respectively).
+### 5. Run the new code
 
-#### More Info
+```bash
+$ nf query run qp-789ghi
+Running query program qp-789ghi...
+Results:
++----------+--------+-------------+
+| Month    | Region | Total Sales |
++----------+--------+-------------+
+| Jan 2025 | North  | $245,678.90 |
+| Jan 2025 | South  | $198,432.45 |
+| Jan 2025 | East   | $312,567.80 |
+| Jan 2025 | West   | $276,543.21 |
+| Feb 2025 | North  | $267,890.12 |
+| Feb 2025 | South  | $210,987.65 |
+...
+```
 
-Minikube will start a local Kubernetes server to deploy the Infactory framework and is essentially a mini Kubernetes cluster running on your local environment - very useful. To get going:
+### 6. Publish the query program
 
 ```bash
-minikube config set driver docker # first time only, make sure you have Docker installed. You can confirm using `minikube config view`
-minikube start
-minikube addons enable ingress
-kubectl config use-context minikube
-kubectl config set-context --current --namespace=dev
-kubectl cluster-info
->>>
-Kubernetes control plane is running at https://127.0.0.1:57094
-CoreDNS is running at https://127.0.0.1:57094/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+$ nf query publish qp-789ghi
+Publishing query program qp-789ghi...
+Query program published successfully!
+Endpoint URL: https://i7y.dev/v1/live/monthly-sales/v1/data
 ```
 
-To stop the Minikube and any containers within the Minikube Kubernetes environment, simply `minikube stop`. You can then delete all Kubernetes clusters in Minikube by executing `minikube delete --all`.
+### 7. Display the available endpoints
+
+```bash
+$ nf endpoints list --project-id my-project-id
++-------------+-----------------+----------------------------------+--------+
+| Endpoint ID | Name            | URL                              | Method |
++-------------+-----------------+----------------------------------+--------+
+| ep-123abc   | Monthly Sales   | /v1/live/monthly-sales/v1/data   | GET    |
+| ep-456def   | Product Details | /v1/live/product-details/v1/data | GET    |
+| ep-789ghi   | Customer Stats  | /v1/live/customer-stats/v1/data  | GET    |
++-------------+-----------------+----------------------------------+--------+
+```
 
-The Minikube and Infactory Kubernetes environments are managed using [namespaces](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/), which we use the same naming scheme and deployment structure as the cloud environments managed by Terraform: `dev`, `staging` and `prod`. To set and see the existing workspace set:
+### 8. Generate a CURL request for an endpoint
 
 ```bash
-kubectl config set-context --current --namespace=<insert-namespace-name-here> # set the new namespace, e.g. dev, staging or prod
-kubectl config view --minify | grep namespace: # validate new workspace
+$ nf endpoints curl-example ep-123abc
+CURL example for endpoint ep-123abc:
+
+curl -X GET "https://i7y.dev/v1/live/monthly-sales/v1/data" \
+  -H "Authorization: Bearer YOUR_API_KEY" \
+  -H "Content-Type: application/json"
 ```
 
-Note that while minikube is running, the Kubernetes `context` is set to Minikube.
+## Python SDK Examples
 
-## Contributing
+The Infactory Python SDK provides a simple interface to interact with the Infactory platform.
 
-Please use the Linear issue tracker to report any erroneous behavior in need of fixes or desired feature requests.
+### 1. Import and setup
 
-If you would like to contribute to development, branch off of `dev` and make contributions to a branch which corresponds to an open issue in [Linear](https://linear.app/infactoryio/team/TECH/all). In the Linear user interface, copy the git branch name, e.g. `tech-103-create-readme`:
+```python
+import infactory as nf
 
-![linear_copy_git_branch_name](/docs/images/linear_copy_git_branch_name.png)
+# Initialize the client with your API key (or it will use NF_API_KEY environment variable)
+client = nf.Client(api_key="your_api_key_here")
+client.connect()
+```
 
-When integrating fixes and features, open a pull request so that code contributions can be reviewed, tested, and documented using both human reviewers (developers and engineers) and automated checks (continuous integration).
+### 2. Create and configure a datasource
 
-### Commit Messages and Releases
+```python
+# Get the current project or set a specific one
+project = client.projects.get("my-project-id")
 
-Your commit messages are important - here's why.
+# Create a new datasource
+datasource = client.datasources.create(
+    name="Product Database",
+    project_id=project.id,
+    type="postgres"
+)
 
-Infactory leverages [release-please](https://github.com/googleapis/release-please-action) to help automate the release process using the [Conventional Commits](https://www.conventionalcommits.org/) specification. When pull requests are opened to the `main` branch, release-please will collate the git commit messages and prepare an organized changelog and release notes. This process can be completed because of the Conventional Commits specification.
+# Configure the datasource connection
+datasource = client.datasources.update(
+    datasource.id,
+    uri="postgresql://username:password@host:port/database"
+)
 
-Conventional Commits provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of. This convention dovetails with SemVer, by describing the features, fixes, and breaking changes made in commit messages. You can check out examples [here](https://www.conventionalcommits.org/en/v1.0.0/#examples). Make a best effort to use the specification when contributing to Infactory code as it dramatically eases the documentation around releases and their features, breaking changes, bug fixes and documentation updates.
+print(f"Datasource created: {datasource.name} (ID: {datasource.id})")
+```
 
-### Environment Setup
+### 3. List datalines in the project
 
-1. First, install, setup and ensure the [Docker](https://www.docker.com/) engine is running on your local machine.
-The Infactory platform runs as a containerized unit.
-2. Python is used as part of the build process for the Docker container(s), so ensure Python is installed on your machine (versions 3.8 - 3.12).
-3. Create an ssh key to use for pulling Infactory code: `ssh-keygen -t rsa -b 4096 -f ~/.ssh/infactory-A`. Then `cd ~/.ssh`.
-4. `chmod 600 infactory-A`
-5. `chmod 644 infactory-A.pub`
-6. Create the config file `vim config`:
+```python
+# Get all datalines for the project
+datalines = client.datalines.list(project_id=project.id)
 
-    ```bash
-    Host *
-    AddKeysToAgent yes
-    UseKeychain yes # remove this line if on Linux and not on MacOS
-    IdentityFile ~/.ssh/infactory-A
-    ```
+for dl in datalines:
+    print(f"Dataline: {dl.name} (ID: {dl.id})")
+```
 
-7. Copy the contents of `cat infactory-A.pub` into Github under >> `Settings` >> `SSH and GPG keys` >> `New SSH key`.
+### 4. Create a query program
+
+```python
+# Choose a dataline from the list
+dataline = datalines[0]
+
+# Create a new query program
+query_program = client.query_programs.create(
+    name="Monthly Sales by Region",
+    question="What are the monthly sales by region?",
+    code="""
+    import pandas as pd
+    
+    def execute(df):
+        # Group by month and region, sum sales
+        result = df.groupby(['month', 'region'])['sales'].sum().reset_index()
+        return result
+    """,
+    dataline_id=dataline.id,
+    project_id=project.id
+)
+
+print(f"Query program created: {query_program.name} (ID: {query_program.id})")
+```
 
-8. Setup and activate the Python virtual environment: `python3 -m venv venv` and `source venv/bin/activate`
-9. Install the project requirements by navigating to `py/infactory` and running `pip install .`.
-10. Git LFS should be installed per the instructions [here](https://git-lfs.com/).
+### 5. Run the query program
 
-### Tests
+```python
+# Evaluate the query program
+result = client.query_programs.evaluate(query_program.id)
 
-When contributing, please ensure to run unit tests and add additional tests as necessary if adding new functionality. To run the tests and obtain test code coverage statistics, run the following from the root directory:
+print("Query results:")
+print(result)
+```
 
-```bash
-pytest --cov=infactory --cov=infactory_api -s -v
-coveralls # to submit the report to https://coveralls.io/github/infactory-io/dev
+### 6. Publish the query program
+
+```python
+# Publish the query program to make it available as an API
+published_program = client.query_programs.publish(query_program.id)
+
+print(f"Query program published: {published_program.id}")
+```
+
+### 7. Create an API endpoint
+
+```python
+# Get API endpoints from the project
+apis = client.projects.get_apis(project.id)
+
+if apis:
+    api = apis[0]
+    print(f"Using existing API: {api.name}")
+else:
+    # Create a new API
+    api = client.apis.create(
+        name="Sales Analytics API",
+        description="API for sales data analytics",
+        project_id=project.id
+    )
+    print(f"Created new API: {api.name}")
+
+# Create an endpoint for the query program
+endpoint = client.apis.create_endpoint(
+    api_id=api.id,
+    name="Monthly Sales",
+    http_method="GET",
+    description="Get monthly sales data by region",
+    queryprogram_id=query_program.id
+)
+
+print(f"Endpoint created: {endpoint.path}")
 ```
 
-You may need to install the Infactory framework in your virtual environment before running the tests along with Poetry:
+### 8. Get endpoint URL and details
 
-```bash
-poetry install
-poetry run pytest --cov=infactory --cov=infactory_api -s -v
+```python
+# Get details for all endpoints in the API
+endpoints = client.apis.get_endpoints(api.id)
+
+for ep in endpoints:
+    print(f"Endpoint: {ep.name}")
+    print(f"URL: https://i7y.dev/v1/live/{api.slug}/{api.version}/{ep.path}")
+    print(f"Method: {ep.http_method}")
+    print(f"Description: {ep.description}")
+    print("-" * 50)
 ```
 
-In addition to being able to execute the tests locally using the above command, Github workflows are executed on the `main` and `dev` branches to ensure that the unit tests pass successfully. See the [coveralls.io](https://coveralls.io/github/infactory-io/dev?branch=main) report to see an interactive unit test code coverage report.
+## Advanced Features
 
-### Versioning
+Check out our [documentation](https://docs.infactory.ai) for more information on advanced features such as:
 
-[Semantic versioning](http://semver.org/) is used for this project. If contributing, please conform to semantic
-versioning guidelines. Future versioning may be handled by tools such as [release-please](https://github.com/googleapis/release-please-action).
+- Custom transformations and data processing
+- Automated data quality checks
+- Integration with ML models
+- Real-time data streaming
+- Team collaboration features
 
-## Deploying
+## Support
 
-See `deploy/readme.md`.
+If you need help or have any questions, please contact us at [email protected] or visit our [support portal](https://support.infactory.ai).