Skip to content

fix: allow IN queries on __key__ #1085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
ad3d2ea
Change to try to ignore __key__ as it is redundant
danieljbruce Mar 10, 2023
4f0db50
Add a check for isDsKey
danieljbruce Mar 14, 2023
23c3200
Revert "Add a check for isDsKey"
danieljbruce Mar 14, 2023
d279595
Add comments so that key is never reassigned
danieljbruce Mar 14, 2023
ee08623
Revert "Add comments so that key is never reassigned"
danieljbruce Mar 14, 2023
72acbf1
Use global entity to test
danieljbruce Mar 14, 2023
258c478
clean up the property class
danieljbruce Mar 14, 2023
f3ff52f
Inline encodeValue
danieljbruce Mar 14, 2023
d91b8bd
Add a test for supporting key with IN
danieljbruce Mar 14, 2023
6337f63
Modifies the query proto to correct the test
danieljbruce Mar 14, 2023
a9bcc0a
Remove comments and clean up test
danieljbruce Mar 14, 2023
7910b43
Use global entity in the new test instead of entit
danieljbruce Mar 15, 2023
f186ee2
Move the query proto into the test
danieljbruce Mar 15, 2023
3e7f057
PR updates
danieljbruce Mar 15, 2023
c6ac45e
Added a system test to validate __key__ with array
danieljbruce Mar 17, 2023
1dcc17f
use unknown instead of any for value
danieljbruce Mar 17, 2023
de1f070
Introduce restricted value when key is used
danieljbruce Mar 17, 2023
25575ba
Add the restricted value type to query too
danieljbruce Mar 17, 2023
8b07e55
Modify the interface for filter
danieljbruce Mar 20, 2023
40a5e19
rename to allowedFilterValueType
danieljbruce Mar 20, 2023
cf9105c
Use capital letter to follow name conventions
danieljbruce Mar 20, 2023
a551ebe
type guard should check instance of
danieljbruce Mar 20, 2023
8bbc2dc
Merge branch 'main' into 1065-query-filter-for-IN-with-keys-test
danieljbruce Mar 21, 2023
d68506e
Merge branch 'main' into 1065-query-filter-for-IN-with-keys-test
danieljbruce Mar 27, 2023
fb9bdf7
Merge branch 'main' into 1065-query-filter-for-IN-with-keys-test
danieljbruce Mar 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 16 additions & 28 deletions src/filter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

import {Operator, Filter as IFilter} from './query';
import {entity} from './entity';
import Key = entity.Key;

const OP_TO_OPERATOR = new Map([
['=', 'EQUAL'],
Expand Down Expand Up @@ -56,42 +57,34 @@ export abstract class EntityFilter {
abstract toProto(): any;
}

export type AllowedFilterValueType<T> = T extends '__key__'
? Key | Key[]
: unknown;

/**
* A PropertyFilter is a filter that gets applied to a query directly.
*
* @see {@link https://cloud.google.com/datastore/docs/concepts/queries#property_filters| Property filters Reference}
*
* @class
*/
export class PropertyFilter extends EntityFilter implements IFilter {
name: string;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
val: any;
op: Operator;

export class PropertyFilter<T extends string>
extends EntityFilter
implements IFilter
{
/**
* Build a Property Filter object.
*
* @param {string} Property
* @param {Operator} operator
* @param {any} val
*/
constructor(property: string, operator: Operator, val: any) {
constructor(
public name: T,
public op: Operator,
public val: AllowedFilterValueType<T>
) {
super();
this.name = property;
this.op = operator;
this.val = val;
}

private encodedValue(): any {
// eslint-disable-next-line @typescript-eslint/no-explicit-any
let value: any = {};
if (this.name === '__key__') {
value.keyValue = entity.keyToKeyProto(this.val);
} else {
value = entity.encodeValue(this.val, this.name);
}
return value;
}

/**
Expand All @@ -100,18 +93,13 @@ export class PropertyFilter extends EntityFilter implements IFilter {
*/
// eslint-disable-next-line
toProto(): any {
const value = new PropertyFilter(
this.name,
this.op,
this.val
).encodedValue();
return {
propertyFilter: {
property: {
name: this.name,
},
op: OP_TO_OPERATOR.get(this.op),
value,
value: entity.encodeValue(this.val, this.name),
},
};
}
Expand Down Expand Up @@ -156,5 +144,5 @@ class CompositeFilter extends EntityFilter {
}

export function isFilter(filter: any): filter is EntityFilter {
return (filter as EntityFilter).toProto !== undefined;
return filter instanceof EntityFilter;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why does this have to change?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggested the fix so I'll answer.

filter.toProto !== undefined only checks if the object has a toProto properties (not set to undefined)
It is the case that EntityFilters have a toProto methods but it is not necessarily the case that an object with a toProto property is an EntityFilter.

filter instanceof EntityFilter is the correct check for filter is EntityFilter

I hope that makes sense.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @vicb, both using instanceof and avoiding type assertions (filter as EntityFilter) are recommended in the Google TypeScript Style Guide. LGTM 👍

}
24 changes: 16 additions & 8 deletions src/query.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import arrify = require('arrify');
import {Key} from 'readline';
import {Datastore} from '.';
import {Entity} from './entity';
import {EntityFilter, isFilter} from './filter';
import {EntityFilter, isFilter, AllowedFilterValueType} from './filter';
import {Transaction} from './transaction';
import {CallOptions} from 'google-gax';
import {RunQueryStreamOptions} from '../src/request';
Expand Down Expand Up @@ -207,12 +207,20 @@ class Query {
* const keyQuery = query.filter('__key__', key);
* ```
*/
filter(propertyOrFilter: string | EntityFilter, value?: {} | null): Query;
filter(propertyOrFilter: string, operator: Operator, value: {} | null): Query;
filter(
propertyOrFilter: string | EntityFilter,
operatorOrValue?: Operator,
value?: {} | null
filter(filter: EntityFilter): Query;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This signature change makes the input parameters more restrictive. Do we really want to go through with this? If so would this be a major version bump?

@vicb @bcoe

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My take is that the old signature was buggy and wouldn't catch unsupported calls. So I see this as a bug fix.

Also I think the impl could be improved by switching on arguments.length. 1, 2, 3 corresponding to the first 3 signatures in order.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do this in a separate issue:

#1096

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to go with a patch version bump

filter<T extends string>(
property: T,
value: AllowedFilterValueType<T>
): Query;
filter<T extends string>(
property: T,
operator: Operator,
value: AllowedFilterValueType<T>
): Query;
filter<T extends string>(
propertyOrFilter: T | EntityFilter,
operatorOrValue?: Operator | AllowedFilterValueType<T>,
value?: AllowedFilterValueType<T>
): Query {
if (isFilter(propertyOrFilter)) {
this.entityFilters.push(propertyOrFilter);
Expand All @@ -223,7 +231,7 @@ class Query {
);
let operator = operatorOrValue as Operator;
if (arguments.length === 2) {
value = operatorOrValue as {};
value = operatorOrValue as AllowedFilterValueType<T>;
operator = '=';
}

Expand Down
34 changes: 34 additions & 0 deletions system-test/datastore.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import {google} from '../protos/protos';
import {Storage} from '@google-cloud/storage';
import {AggregateField} from '../src/aggregate';
import {PropertyFilter, EntityFilter, and, or} from '../src/filter';
import {entity} from '../src/entity';
import KEY_SYMBOL = entity.KEY_SYMBOL;

describe('Datastore', () => {
const testKinds: string[] = [];
Expand Down Expand Up @@ -794,6 +796,38 @@ describe('Datastore', () => {
assert.strictEqual(entities!.length, 3);
});

it('should filter queries with __key__ and IN', async () => {
const key1 = datastore.key(['Book', 'GoT', 'Character', 'Rickard']);
const key2 = datastore.key([
'Book',
'GoT',
'Character',
'Rickard',
'Character',
'Eddard',
'Character',
'Sansa',
]);
const key3 = datastore.key([
'Book',
'GoT',
'Character',
'Rickard',
'Character',
'Eddard',
]);
const value = [key1, key2, key3];
const q = datastore
.createQuery('Character')
.hasAncestor(ancestor)
.filter('__key__', 'IN', value);
const [entities] = await datastore.runQuery(q);
assert.strictEqual(entities!.length, 3);
assert.deepStrictEqual(entities[0][KEY_SYMBOL], key1);
assert.deepStrictEqual(entities[1][KEY_SYMBOL], key3);
assert.deepStrictEqual(entities[2][KEY_SYMBOL], key2);
});

it('should filter queries with NOT_IN', async () => {
const q = datastore
.createQuery('Character')
Expand Down
62 changes: 58 additions & 4 deletions test/entity.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ import {beforeEach, afterEach, describe, it} from 'mocha';
import * as extend from 'extend';
import * as sinon from 'sinon';
import {Datastore} from '../src';
import {Entity} from '../src/entity';
import {Entity, entity as globalEntity} from '../src/entity';
import {IntegerTypeCastOptions} from '../src/query';
import {PropertyFilter, EntityFilter, and} from '../src/filter';

Expand Down Expand Up @@ -1873,7 +1873,7 @@ describe('entity', () => {
};

it('should support all configurations of a query', () => {
const ancestorKey = new entity.Key({
const ancestorKey = new globalEntity.Key({
path: ['Kind2', 'somename'],
});

Expand All @@ -1894,8 +1894,62 @@ describe('entity', () => {
assert.deepStrictEqual(entity.queryToQueryProto(query), queryProto);
});

it('should support using __key__ with array as value', () => {
const keyWithInQuery = {
distinctOn: [],
filter: {
compositeFilter: {
filters: [
{
propertyFilter: {
op: 'IN',
property: {
name: '__key__',
},
value: {
arrayValue: {
values: [
{
keyValue: {
path: [
{
kind: 'Kind1',
name: 'key1',
},
],
},
},
],
},
},
},
},
],
op: 'AND',
},
},
kind: [
{
name: 'Kind1',
},
],
order: [],
projection: [],
};

const ds = new Datastore({projectId: 'project-id'});

const query = ds
.createQuery('Kind1')
.filter('__key__', 'IN', [
new globalEntity.Key({path: ['Kind1', 'key1']}),
]);

assert.deepStrictEqual(entity.queryToQueryProto(query), keyWithInQuery);
});

it('should support the filter method with Filter objects', () => {
const ancestorKey = new entity.Key({
const ancestorKey = new globalEntity.Key({
path: ['Kind2', 'somename'],
});

Expand All @@ -1916,7 +1970,7 @@ describe('entity', () => {
});

it('should support the filter method with AND', () => {
const ancestorKey = new entity.Key({
const ancestorKey = new globalEntity.Key({
path: ['Kind2', 'somename'],
});

Expand Down