Skip to content

Switch out old APIs #259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 70 commits into from
Jan 31, 2025
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
4e24685
add todo
ken-zlai Jan 22, 2025
7e42a2f
add JoinRequest
ken-zlai Jan 23, 2025
649e69c
implement a new json serializer which handles unions properly
ken-zlai Jan 23, 2025
d3c10c7
create ConfHandler, implement getJoin
ken-zlai Jan 23, 2025
3c86580
add getJoin to frontend
ken-zlai Jan 23, 2025
4b59d12
add /thrift route for testing purposes
ken-zlai Jan 23, 2025
4b37045
remove todo
ken-zlai Jan 23, 2025
c7658d1
add new Join() explanation
ken-zlai Jan 23, 2025
ec318f1
add function explanation comment
ken-zlai Jan 23, 2025
c424a54
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 23, 2025
f7b52f3
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 23, 2025
cefde52
add conf thrift
ken-zlai Jan 23, 2025
c08151a
implement getConf (not fully tested)
ken-zlai Jan 23, 2025
124e9ff
frontend api calls for model, groupby, staging query
ken-zlai Jan 23, 2025
c5d111e
use LogicalNode type
ken-zlai Jan 23, 2025
7b534cc
implement /api/v1/conf/list
ken-zlai Jan 23, 2025
e581d3e
frontend api stuff
ken-zlai Jan 23, 2025
9425487
similar spacing
ken-zlai Jan 23, 2025
a3cf764
get a groupby and model that exist
ken-zlai Jan 24, 2025
7c981b7
use ConfType
ken-zlai Jan 24, 2025
3ca3a09
change routes
ken-zlai Jan 24, 2025
7fd4fbb
create LogicalNodeTable
ken-zlai Jan 24, 2025
c29fedd
show joins/models/staging queries/groupbys on their page
ken-zlai Jan 24, 2025
bdd3858
just use table directly, its simpler
ken-zlai Jan 24, 2025
09b0dbc
empty state for table
ken-zlai Jan 24, 2025
4686e94
put back logic to only make risk.user_transactions.txn_join clickable
ken-zlai Jan 24, 2025
dc1a110
fix npm run check
ken-zlai Jan 24, 2025
cf5c1ad
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 27, 2025
3627e81
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 27, 2025
d183759
create JoinDriftRequest and JoinDriftResponse
ken-zlai Jan 28, 2025
cf440e2
implement drift handler
ken-zlai Jan 28, 2025
8868ed8
getJoinDrift on the frontend
ken-zlai Jan 28, 2025
9d90866
add featureName to JoinDriftRequest
ken-zlai Jan 29, 2025
e2bede7
frontend endpoint for featureDrift
ken-zlai Jan 29, 2025
f06f461
add hub endpoint: /api/v1/join/:name/feature/:featureName/drift
ken-zlai Jan 29, 2025
df5ea52
implement getFeatureDrift and filtering
ken-zlai Jan 29, 2025
1201606
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 29, 2025
3db534f
simplify some duplicate code
ken-zlai Jan 29, 2025
1f3afb3
scalafmt
ken-zlai Jan 29, 2025
78913d9
modify search endpoint to return ConfListResponse, also search thru j…
ken-zlai Jan 29, 2025
69008fc
frontend changes for new search
ken-zlai Jan 29, 2025
ca7047e
comments :)
ken-zlai Jan 29, 2025
4b176d1
put back old error code
ken-zlai Jan 29, 2025
05a5520
confhandler tests
ken-zlai Jan 29, 2025
b837620
delete old code
ken-zlai Jan 29, 2025
3e05c32
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 29, 2025
eef46a1
sbt scalafixAll, add logs
ken-zlai Jan 29, 2025
bf58d8c
use new serializer each time
ken-zlai Jan 30, 2025
224a894
add magicNullLong
ken-zlai Jan 30, 2025
a6b7b56
add JoinSummaryRequest
ken-zlai Jan 30, 2025
7d55036
create summary endpoint
ken-zlai Jan 30, 2025
b723b59
implement getFeatureSummary
ken-zlai Jan 30, 2025
b7edf22
call api from frontend
ken-zlai Jan 30, 2025
6ac85b3
fmt and fix
ken-zlai Jan 30, 2025
e8a7686
change magic values to safe value
ken-zlai Jan 30, 2025
8d59fec
oops put search endpoint back
ken-zlai Jan 31, 2025
70f1614
Drift handler test
ken-zlai Jan 31, 2025
21b1ef5
column -> feature
ken-zlai Jan 31, 2025
8cadcd4
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 31, 2025
bbeb642
logger.info -> logger.debug
ken-zlai Jan 31, 2025
e352adc
fix and fmt
ken-zlai Jan 31, 2025
d288225
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 31, 2025
46efc2f
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 31, 2025
d01fd09
Merge branch 'main' into ken/thrift-observability-apis
ken-zlai Jan 31, 2025
431fc7b
remove getModels
ken-zlai Jan 31, 2025
f6d848e
more resiliant code for TimeSeriesHandler
ken-zlai Jan 31, 2025
eb232f7
remove old import
ken-zlai Jan 31, 2025
6aca8a5
delete unused getJoins
ken-zlai Jan 31, 2025
7b21828
cleaner code for routing stuff
ken-zlai Jan 31, 2025
a18fb90
remove unused
ken-zlai Jan 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions api/thrift/api.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -432,4 +432,8 @@ struct Model {
3: optional TDataType outputSchema
4: optional Source source
5: optional map<string, string> modelParams
}

struct JoinRequest {
1: optional string name
}
10 changes: 9 additions & 1 deletion frontend/src/lib/api/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import type {
JoinTimeSeriesResponse,
ModelsResponse
} from '$lib/types/Model/Model';
import type { Join } from '$lib/types/codegen';

export type ApiOptions = {
base?: string;
Expand Down Expand Up @@ -43,6 +44,13 @@ export class Api {
return this.#send<JoinsResponse>(`joins?${params.toString()}`);
}

async getJoin(name: string): Promise<Join> {
// ideally, we would do this so join instanceof Join === true. however, new Join() will not work until the sample data is updated w/ compile.py to match the latest thrift definition
// const response = await this.#send<Join>(`join/${name}`);
// return Object.assign(new Join(), response);
return this.#send<Join>(`join/${name}`);
}

async search(term: string, limit: number = 20) {
const params = new URLSearchParams({
term,
Expand Down Expand Up @@ -151,7 +159,7 @@ export class Api {
} else {
const text = (await response.text?.()) ?? '';
console.error(`Failed request: "${text}" for url: ${url}`);
error(response.status);
throw error(response.status);
}
});
}
Expand Down
14 changes: 14 additions & 0 deletions frontend/src/routes/thrift/+page.svelte
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<!-- todo: this is only for testing, will be deleted before merge -->
<script lang="ts">
import { Api } from '$lib/api/api';

$effect(() => {
const api = new Api({ fetch });

api.getJoin('sample_team.sample_chaining_join.parent_join').then((join) => {
console.log(join);
});
});
</script>

<div>hello!</div>
5 changes: 4 additions & 1 deletion hub/src/main/java/ai/chronon/hub/HubVerticle.java
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
package ai.chronon.hub;

import ai.chronon.api.Constants;
import ai.chronon.api.JoinRequest;
import ai.chronon.hub.handlers.*;
import ai.chronon.hub.store.MonitoringModelStore;
import ai.chronon.online.Api;
import ai.chronon.online.KVStore;
import ai.chronon.online.stats.DriftStore;
import ai.chronon.service.ApiProvider;
import ai.chronon.service.ConfigStore;
import ai.chronon.service.RouteHandlerWrapper;
import ai.chronon.spark.utils.InMemoryKvStore;
import io.vertx.core.AbstractVerticle;
import io.vertx.core.Promise;
Expand Down Expand Up @@ -56,8 +58,9 @@ protected void startHttpServer(int port, String configJsonString, Api api, Promi

// Add routes for metadata retrieval
MonitoringModelStore store = new MonitoringModelStore(api);
ConfHandler confHandler = new ConfHandler(store);
router.get("/api/v1/models").handler(new ModelsHandler(store));
router.get("/api/v1/join/:name").handler(new JoinsHandler(store).getHandler());
router.get("/api/v1/join/:name").handler(RouteHandlerWrapper.createHandler(confHandler::getJoin, JoinRequest.class));
router.get("/api/v1/joins").handler(new JoinsHandler(store).listHandler());
router.get("/api/v1/search").handler(new SearchHandler(store));

Expand Down
23 changes: 23 additions & 0 deletions hub/src/main/scala/ai/chronon/hub/handlers/ConfHandler.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package ai.chronon.hub.handlers

import ai.chronon.api.{JoinRequest, Join}
import ai.chronon.hub.store.MonitoringModelStore
import org.slf4j.{Logger, LoggerFactory}

class ConfHandler(store: MonitoringModelStore) {
private val logger: Logger = LoggerFactory.getLogger(this.getClass)

/**
* Returns a specific join by name (/api/v1/join/:name)
*/
def getJoin(req: JoinRequest): Join = {
logger.debug("Retrieving {}", req.getName);
val joins = store.configRegistryCache("default").joins
joins
.find(_.getMetaData.getName.equalsIgnoreCase(req.getName))
.getOrElse(throw new RuntimeException(s"Unable to retrieve ${req.getName}"))
}

// todo getJoinNames, listNames, etc
// todo all others (groupBy, model, observability etc)
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import ai.chronon.api.thrift.*;
import ai.chronon.api.thrift.protocol.TBinaryProtocol;
import ai.chronon.api.thrift.protocol.TSimpleJSONProtocol;
import ai.chronon.api.thrift.transport.TTransportException;
import io.vertx.core.Handler;
import io.vertx.core.json.JsonObject;
Expand Down Expand Up @@ -49,7 +50,13 @@ public class RouteHandlerWrapper {

private static final ThreadLocal<Base64.Encoder> base64Encoder = ThreadLocal.withInitial(Base64::getEncoder);
private static final ThreadLocal<Base64.Decoder> base64Decoder = ThreadLocal.withInitial(Base64::getDecoder);

private static final ThreadLocal<TSerializer> jsonSerializer = ThreadLocal.withInitial(() -> {
try {
return new TSerializer(new TSimpleJSONProtocol.Factory());
} catch (TTransportException e) {
throw new RuntimeException(e);
}
});

public static <T extends TBase> T deserializeTBinaryBase64(String base64Data, Class<? extends TBase> clazz) throws NoSuchMethodException, InvocationTargetException, InstantiationException, IllegalAccessException, TException {
byte[] binaryData = base64Decoder.get().decode(base64Data);
Expand Down Expand Up @@ -87,8 +94,19 @@ public static <I, O> Handler<RoutingContext> createHandler(Function<I, O> transf

String responseFormat = ctx.request().getHeader(RESPONSE_CONTENT_TYPE_HEADER);
if (responseFormat == null || responseFormat.equals("application/json")) {
// Send json response
ctx.response().setStatusCode(200).putHeader("content-type", JSON_TYPE_VALUE).end(JsonObject.mapFrom(output).encode());
try {
String jsonString = jsonSerializer.get().toString((TBase)output);
ctx.response()
.setStatusCode(200)
.putHeader("content-type", JSON_TYPE_VALUE)
.end(jsonString);
} catch (TException e) {
LOGGER.error("Failed to serialize response", e);
throw new RuntimeException(e);
} catch (Exception e) {
LOGGER.error("Unexpected error during serialization", e);
throw new RuntimeException(e);
}
} else {
if (!responseFormat.equals(TBINARY_B64_TYPE_VALUE)) {
throw new IllegalArgumentException(String.format("Unsupported response-content-type: %s. Supported values are: %s and %s", responseFormat, JSON_TYPE_VALUE, TBINARY_B64_TYPE_VALUE));
Expand Down
Loading